Data lakehouse system and data processing method based on data lakehouse system

Abstract

Disclosed in the present disclosure are a data lakehouse system and a data processing method based on the data lakehouse system. The data lakehouse system comprises: a data exploration layer used for accessing a target data source, constructing a first user-defined function and configuring a target data pipeline, wherein the target data source comprises original data information to be processed, and the target data pipeline calls the first user-defined function to process the original data information; a data lakehouse used for executing the target data pipeline so as to process the original data information to obtain first data information, and further used for storing the first data information; and a data application layer, wherein a first object calls the first data information by means of the data application layer. The present disclosure solves the technical problem in the related art of low data processing efficiency caused by the fact that the data lakehouse requires a user to manually configure information such as data sources, table structures and resources before processing and storing data.

Description

[0001] This disclosure claims priority to Chinese patent application number 202410154728.3, filed with the Patent Office of China on February 2, 2024, entitled "Integrated Lake-Warehouse System and Data Processing Method Based on the Integrated Lake-Warehouse System," the entire contents of which are incorporated herein by reference. Technical Field This disclosure relates to the field of data processing technology, and more specifically, to an integrated lake-warehouse system and a data processing method based on the integrated lake-warehouse system. BACKGROUND: During data development in big data analysis / online analytical processing (OLAP) scenarios, existing integrated lake-warehouse systems have drawbacks in data integration, data modeling and development, data analysis, data governance, and data services. For example, if data sources are required and data pipeline configuration is required, users must configure various information, such as data source / target system / connector configuration, ETL (Extract-Transform-Load) configuration, and resource / network configuration. This manual resource management and data validation by users impacts the efficiency of data processing in the integrated lake-warehouse system. Currently, no effective solution has been proposed to address the problem that the aforementioned related art integrated lake-warehouse systems require users to manually configure information such as data sources, table structures, and resources before processing and storing data, resulting in relatively low data processing efficiency. SUMMARY OF THE INVENTION: The present disclosure provides an integrated lake-warehouse system and a data processing method based on the integrated lake-warehouse system to at least address the technical problem in the related art that the integrated lake-warehouse system requires users to manually configure information such as data sources, table structures, and resources before processing and storing data, resulting in relatively low data processing efficiency. According to one aspect of an embodiment of the present disclosure, a lake-warehouse integrated system is provided, including: a data exploration layer, wherein the data exploration layer is used to access a target data source, construct a first custom function, and configure a target data pipeline, the target data source includes raw data information to be processed, and the target data pipeline calls the first custom function to process the raw data information; a lake-warehouse integrated system, wherein the lake-warehouse integrated system is used to execute the target data pipeline to process the raw data information to obtain first data information, and the lake-warehouse integrated system is also used to store the first data information; a data application layer, wherein the first object calls the first data information through the data application layer.Furthermore, the data exploration layer includes: a data source access module, wherein the data source access module accesses the target data source based on the requirements of the second object; a function construction module, wherein the function construction module constructs the first custom function based on the processing requirements of the raw data information; and a data pipeline configuration module, wherein the data pipeline configuration module configures the target data pipeline based on the first custom function and the processing requirements so that the target data pipeline is executed in the lake-warehouse integration. Furthermore, the lake-warehouse integration also includes: a raw data layer, wherein the raw data layer is used to store raw data information imported from the target data source; a pipeline layer, wherein the target data pipeline is executed in the pipeline layer to process the raw data information to obtain the first data information, wherein the first data information is the data information required by the second object; and a lake-warehouse integration layer, wherein the lake-warehouse integration layer is used to store the first data information. Furthermore, the data application layer also includes: a data management module, wherein the data management module is used to manage the first data information; a data query and analysis module, wherein the data query and analysis module is used to query and analyze the first data information to obtain the data information required by the first object; and a data application module, wherein the data application module is used to process and handle the first data information to meet task requirements. Furthermore, the data exploration layer also includes: a metadata processing module, wherein the metadata processing module is used to detect the first data information stored in the lake-warehouse integrated layer to obtain metadata information corresponding to the first data information. According to another aspect of the embodiment of the present disclosure, a data processing method based on a lake-warehouse integrated system is also provided, including: receiving original data information to be processed from a target data source through the lake-warehouse integrated system in the lake-warehouse integrated system; when there is no custom function for processing the original data information in the data exploration layer of the lake-warehouse integrated system, generating the first custom function based on the processing requirements of the original data information through the data exploration layer, and generating a target data pipeline based on the processing requirements and the first custom function, wherein the target data pipeline calls the first custom function to process the original data information; processing the original data information according to the target data pipeline through the lake-warehouse integrated system in the lake-warehouse integrated system to obtain first data information.Furthermore, before receiving the original data information to be processed from the target data source through the lake-warehouse integration in the lake-warehouse integration system, the method also includes: receiving the first information input by the first target object through the first front-end interface corresponding to the data source access module in the data exploration layer, and sending the first information to the data source access module through the first front-end interface, wherein the first information at least includes the identification information of the data source; determining the target data source based on the target language model and the identification information in the first information through the data source access module, and pushing the parameter information corresponding to the target data source to the first front-end interface; receiving the second information input by the first target object through the first front-end interface, and establishing a connection with the target data source based on the second information through the data source access module to receive the original data information to be processed from the target data source, wherein the second information at least includes the identity identification ID information of the first target object. Furthermore, generating the first custom function based on the processing requirements for the raw data information through the data exploration layer in the lake-warehouse integrated system includes: receiving third information input by the first target object through a second front-end interface corresponding to a function construction module in the data exploration layer; sending the third information to the function construction module through the second front-end interface, wherein the third information is a description of the processing requirements; determining, by the function construction module, whether a target tool needs to be called based on a target language model, and obtaining a determination result; if the determination result indicates that the target tool needs to be called, processing the third information based on the target language model and the target tool by the function construction module to obtain the first custom function; and pushing the first custom function to the second front-end interface by the function construction module. Furthermore, after obtaining the first custom function, the method further includes: obtaining a preset custom function specification through the function construction module; verifying the first custom function according to the custom function specification through the function construction module to obtain a verification result; and if the verification result indicates that there are no abnormalities in the first custom function, compiling and deploying the first custom function through the function construction module so that the target data pipeline can call the first custom function for data processing.Furthermore, after the data exploration layer in the lake-warehouse integrated system generates the first custom function based on the processing requirements for the raw data information, the method further includes: calling a node-based workflow automation tool through the function construction module in the data exploration layer to compile the first custom function to obtain a compiled first custom function; constructing a workflow based on the compiled first custom function through the function construction module, wherein the workflow is used to verify whether the first custom function meets the processing requirements; executing the workflow through the function construction module to obtain an execution result, and verifying whether the first custom function meets the processing requirements based on the execution result. Furthermore, generating a target data pipeline based on the processing requirements and the first custom function includes: determining a data pipeline template based on a target language model and the processing requirements through the data pipeline configuration module in the data exploration layer; configuring the data pipeline based on the data pipeline template and the first custom function through the data pipeline configuration module to obtain configuration information; and generating the target data pipeline based on the configuration information through the data pipeline configuration module. Furthermore, after receiving the raw data information to be processed from the target data source through the lake-warehouse integration in the lake-warehouse integration system, the method further includes: determining the data source type of the target data source through the lake-warehouse integration; determining a write operator through the lake-warehouse integration based on the data source type; and writing the raw data information into the raw data layer of the lake-warehouse integration system through the lake-warehouse integration based on the write operator. Furthermore, after the lake-warehouse integration in the lake-warehouse integration system processes the raw data information based on the target data pipeline to obtain first data information, the method further includes: receiving a data query request initiated by a second target object through a third front-end interface corresponding to the data application layer in the lake-warehouse integration system, and sending the data query request to the data application layer through the third front-end interface; obtaining second data information corresponding to the data query request from the lake-warehouse integration layer of the lake-warehouse integration system through the data application layer based on the data query request; and pushing the second data information to the third front-end interface through the data application layer.Furthermore, after the raw data information is processed by the lake-warehouse integrated system according to the target data pipeline to obtain first data information, the method further includes: constructing a training sample set by the data application layer according to the first data information; and training a neural network model for a target scenario corresponding to the first data information by the data application layer according to the training sample set to obtain a target neural model. According to another aspect of an embodiment of the present disclosure, a data processing device based on the lake-warehouse integrated system is provided, comprising: a first receiving unit for receiving raw data information to be processed from a target data source by the lake-warehouse integrated system in the lake-warehouse integrated system; a generating unit for generating, if a custom function for processing the raw data information does not exist in the data exploration layer in the lake-warehouse integrated system, the first custom function based on a processing requirement for the raw data information by the data exploration layer, and generating a target data pipeline based on the processing requirement and the first custom function, wherein the target data pipeline calls the first custom function to process the raw data information; and a processing unit for processing the raw data information by the lake-warehouse integrated system according to the target data pipeline to obtain first data information. Furthermore, the device also includes: a second receiving unit, which is used to receive the first information input by the first target object through the first front-end interface corresponding to the data source access module in the data exploration layer before receiving the original data information to be processed from the target data source through the lake-warehouse integrated system, and send the first information to the data source access module through the first front-end interface, wherein the first information at least includes the identification information of the data source; a first determination unit, which is used to determine the target data source based on the target language model and the identification information in the first information through the data source access module, and push the parameter information corresponding to the target data source to the first front-end interface; a third receiving unit, which is used to receive the second information input by the first target object through the first front-end interface, and establish a connection with the target data source based on the second information through the data source access module to receive the original data information to be processed from the target data source, wherein the second information at least includes the identity identification ID information of the first target object.Furthermore, the generation unit includes: a receiving module, which is used to receive third information input by the first target object through the second front-end interface corresponding to the function construction module in the data exploration layer, and send the third information to the function construction module through the second front-end interface, wherein the third information is the description information of the processing requirement; a judgment module, which is used to judge whether it is necessary to call the target tool based on the target language model through the function construction module to obtain a judgment result; a processing module, which is used to process the third information based on the target language model and the target tool through the function construction module when the judgment result indicates that the target tool needs to be called, to obtain the first custom function; and a pushing module, which is used to push the first custom function to the second front-end interface through the function construction module. Furthermore, the apparatus further includes: a first acquisition unit for, after obtaining the first custom function, obtaining a preset custom function specification through the function construction module; a verification unit for, through the function construction module, verifying the first custom function according to the custom function specification to obtain a verification result; a first compilation unit for, if the verification result indicates that the first custom function does not have any anomalies, compiling and deploying the first custom function through the function construction module, so that the target data pipeline can call the first custom function for data processing. Furthermore, the apparatus further includes: a second compilation unit for, after generating the first custom function based on the processing requirements of the raw data information through the data exploration layer in the lake-warehouse integrated system, calling a node-based workflow automation tool to compile the first custom function through the function construction module in the data exploration layer to obtain a compiled first custom function; a first construction unit for, through the function construction module, constructing a workflow based on the compiled first custom function, wherein the workflow is used to verify whether the first custom function meets the processing requirements; and an execution unit for executing the workflow through the function construction module, obtaining an execution result, and verifying whether the first custom function meets the processing requirements based on the execution result.Furthermore, the generation unit includes: a determination module for determining a data pipeline template based on a target language model and the processing requirements through a data pipeline configuration module in the data exploration layer; a configuration module for configuring the data pipeline according to the data pipeline template and the first custom function through the data pipeline configuration module to obtain configuration information; and a generation module for generating the target data pipeline according to the configuration information through the data pipeline configuration module. Furthermore, the device also includes: a second determination unit for determining the data source type of the target data source through the lake warehouse integration after receiving the raw data information to be processed from the target data source through the lake warehouse integration in the lake warehouse integration system; a third determination unit for determining a write operator through the lake warehouse integration according to the data source type; and a write unit for writing the raw data information into the raw data layer of the lake warehouse integration system through the lake warehouse integration according to the write operator. Furthermore, the device also includes: a fourth receiving unit, which is used to process the original data information according to the target data pipeline through the lake warehouse integrated system to obtain the first data information, and then receive the data query request initiated by the second target object through the third front-end interface corresponding to the data application layer in the lake warehouse integrated system, and send the data query request to the data application layer through the third front-end interface; a second acquisition unit, which is used to obtain the second data information corresponding to the data query request from the lake warehouse integrated layer of the lake warehouse integrated system through the data application layer according to the data query request; a pushing unit, which is used to push the second data information to the third front-end interface through the data application layer. Further, the device also includes: a second construction unit, which is used to process the original data information according to the target data pipeline through the lake warehouse integrated system to obtain the first data information, and then construct a training sample set based on the first data information through the data application layer; a training unit, which is used to train the neural network model under the target scenario corresponding to the first data information through the data application layer according to the training sample set to obtain a target neural model. According to another aspect of an embodiment of the present disclosure, a computer-readable storage medium is further provided, which stores a program, wherein when the program is running, the device where the storage medium is located is controlled to execute any one of the above-mentioned data processing methods based on the lake-warehouse integrated system.According to another aspect of an embodiment of the present disclosure, an electronic device is provided, comprising a memory storing an executable program; and a processor configured to execute the program, wherein when the program executes, the program executes any one of the aforementioned data processing methods based on a lake-warehouse integrated system. According to another aspect of an embodiment of the present disclosure, a computer program product is provided, comprising a computer program, wherein the computer program is executed by the processor to execute any one of the aforementioned data processing methods based on a lake-warehouse integrated system. In this embodiment of the present disclosure, a data exploration layer is employed, wherein the data exploration layer is configured to access a target data source, construct a first custom function, and configure a target data pipeline. The target data source includes raw data information to be processed, and the target data pipeline calls the first custom function to process the raw data information. A lake-warehouse integrated system is employed, wherein the lake-warehouse integrated system is configured to execute the target data pipeline to process the raw data information to obtain first data information, and is also configured to store the first data information. A data application layer is employed, wherein the first object calls the first data information through the data application layer. This solves the technical problem in related technologies where lake-warehouse integrated systems require users to manually configure information such as data sources, table structures, and resources before processing and storing data, resulting in relatively low data processing efficiency. In this solution, the data exploration layer accesses the target data source, constructs a first custom function, and configures the target data pipeline. The target data pipeline is then executed in the lake-warehouse system to process the raw data and obtain the first data. This eliminates the need for users to manually configure data sources, table structures, and resources for data processing and storage. The data exploration layer allows for rapid access to the target data source, accurate construction of the first custom function, and configuration of the corresponding target data pipeline, thereby improving data processing efficiency. The accompanying drawings described herein are provided to provide a further understanding of the present disclosure and constitute a part thereof. The illustrative embodiments of the present disclosure and their descriptions are provided to explain the present disclosure and are not intended to unduly limit the present disclosure.In the accompanying drawings: Figure 1 is a schematic diagram of the lake-warehouse integrated system provided according to the first embodiment of the present disclosure; Figure 2 is a schematic diagram of the data exploration layer provided according to the first embodiment of the present disclosure; Figure 3 is a schematic diagram of the lake-warehouse integrated system provided according to the first embodiment of the present disclosure; Figure 4 is a schematic diagram of the data application layer provided according to the first embodiment of the present disclosure; Figure 5 is a schematic diagram of the data exploration layer provided according to the first embodiment of the present disclosure; Figure 6 is a schematic diagram of the lake-warehouse integrated system provided according to the first embodiment of the present disclosure; Figure 7 is a flowchart of the data processing method based on the lake-warehouse integrated system provided according to the second embodiment of the present disclosure; Figure 8 is an interface schematic diagram of the data access module provided according to the second embodiment of the present disclosure; Figure 9 is a schematic diagram of the workflow provided according to the second embodiment of the present disclosure; Figure 10 is a processing flow chart of the custom function provided according to the second embodiment of the present disclosure; Figure 11 is a flowchart of the data processing method based on the lake-warehouse integrated system provided according to the second embodiment of the present disclosure; Figure 12 is a flowchart of the data processing method based on the lake-warehouse integrated system provided according to the second embodiment of the present disclosure; Figure 13 is a schematic diagram of the data processing device based on the lake-warehouse integrated system provided according to the third embodiment of the present disclosure; Figure 14 is a schematic diagram of the electronic device provided according to the fourth embodiment of the present disclosure. DETAILED DESCRIPTION To help those skilled in the art better understand the present disclosure, the following will provide a clear and complete description of the technical solutions in the embodiments of the present disclosure, in conjunction with the accompanying drawings. It should be noted that the described embodiments represent only a portion of the present disclosure, and are not exhaustive. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present disclosure without inventive effort should fall within the scope of protection of the present disclosure. It should be noted that the terms "first," "second," and so on, in the specification and claims of the present disclosure, and in the accompanying drawings, are used to distinguish similar objects and are not necessarily used to describe a specific order or precedence. It should be understood that such terms are interchangeable where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than that illustrated or described herein. Furthermore, the terms "including," "having," and any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus comprising a series of steps or units is not necessarily limited to the steps or units expressly listed, but may include other steps or units not expressly listed or inherent to such process, method, product, or apparatus. First, some nouns or terms that appear in the description of the embodiments of the present disclosure are subject to the following explanations:

[0002] Flink: Apache Flink is a distributed stream processing engine written in Java and Scala. It can perform computations and analysis on both real-time streaming and batch data, and is a high-performance, high-availability stream processing framework. Data Lake: A data lake is a system used to store data of various types and formats. Data lakes are typically highly scalable, flexible, and cost-effective. Unlike traditional data warehouses, data lakes do not require predefined data schemas and can accommodate structured, semi-structured, and unstructured data. They can also ingest data from diverse sources, such as sensors, logs, and social media.

[0003] UDF: A user-defined function (UDF) is a programming construct that allows developers to define their own functions based on specific needs and apply them to big data processing frameworks. UDFs extend existing function libraries, enabling developers to process and transform data by writing custom logic. UDFs can be used for a variety of big data processing tasks, such as data cleansing, data transformation, and feature extraction. They provide a flexible way for developers to customize data processing logic based on task requirements, thereby improving the efficiency and accuracy of data analysis and processing. UDFs are typically written in programming languages ​​such as Java and Python and registered and called on the big data platform to apply custom functions during data processing. Data lake integration: A novel open architecture connects data warehouses and data lakes, combining the high performance and management capabilities of data warehouses with the flexibility of data lakes. The underlying layer supports the coexistence of multiple data types and enables data sharing. The upper layer can be accessed through a unified encapsulated interface, supporting both real-time query and analysis. It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, and displayed data, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties. The collection, use, and processing of relevant data must comply with the relevant laws, regulations, and standards of the relevant region, and corresponding operation portals are provided for users to choose to authorize or refuse. Example 1 According to an embodiment of the present disclosure, a lake-warehouse integrated system is also provided. As shown in Figure 1, the lake-warehouse integrated system includes: a data exploration layer 10, a lake-warehouse integration 20, and a data application layer 30. Data exploration layer 10, wherein the data exploration layer is used to access the target data source, construct the first custom function and configure the target data pipeline, the target data source includes the original data information to be processed, and the target data pipeline calls the first custom function to process the original data information; Lake warehouse integration 20, wherein the lake warehouse integration is used to execute the target data pipeline to process the original data information to obtain the first data information, and the lake warehouse integration is also used to store the first data information; Data application layer 30, wherein the first object calls the first data information through the data application layer.Optionally, as shown in Figure 1, the lake-warehouse integrated system provided in the first embodiment of the present disclosure includes a data exploration layer 10, a lake-warehouse integration 20, and a data application layer 30. The data exploration layer 10 intelligently understands user needs and provides a convenient interface to help users better understand and utilize data. The data exploration layer 10 can access data sources based on user needs with one click and supports SQL (Structured Query Language), NoSQL (non-relational databases), queues, and task-customized data formats. The data exploration layer 10 is also used to build a first user-defined function (ML-UDF, Multi-Lingual User Defined Function). UDFs can be developed across programming languages, allowing users to select the most suitable programming language for processing text, rich text, audio, or video data in different scenarios. The data exploration layer 10 provides a variety of multimodal semantics to support multimodal information processing. ML-UDFs can be understood as independent serverless functions and support cloud-native deployment. The data exploration layer 10 is also used for data pipeline configuration. Data pipelines can be configured using Dynamic Script Language (DSL)—a data pipeline DSL is intelligently generated and deployed based on user-described task requirements. Data pipelines can also be configured using SQL: users can create data pipelines using specific SQL syntax. The data exploration layer 10 automatically connects to data sources, builds and defines functions, and configures and deploys data pipelines, eliminating the need for users to submit numerous forms when connecting to data sources and configure cloud vendor-specific resources, networks, and other tasks. The target data pipeline is executed in the Lake-Warehouse Integration 20, processing the raw data information to obtain the first data information. The Lake-Warehouse Integration 20 is also used to store the first data information. The Lake-Warehouse Integration 20 also stores the raw data, providing consistent incremental data stream reading, incremental querying, data expiration, data partitioning, and data replay capabilities for subsequent data pipelines. The lake-warehouse integration 20 processes data based on stream computing and ML-UDF, reading raw data in streams and performing operations such as data processing, data mining, data pre-computation, and data-derived metrics. It then outputs multi-modal data (i.e., the first data information described above). Furthermore, the lake-warehouse integration 20 possesses both lake and warehouse capabilities, supporting data storage in standard lake-warehouse formats and the storage and management of multi-modal data produced by the streaming pipeline layer, providing support for data applications. The data application layer 30 allows users to retrieve required data from the lake-warehouse integration 20 based on actual needs.In summary, the target data source is accessed through the data exploration layer, a first custom function is constructed, and a target data pipeline is configured. The target data pipeline is then executed in the lake-warehouse integration system to process the raw data information and obtain the first data information. This eliminates the need for users to manually configure information such as data sources, table structures, and resources for data processing and storage. The data exploration layer allows for rapid access to the target data source, accurate construction of the first custom function, and configuration of the corresponding target data pipeline, thereby improving data processing efficiency. To achieve rapid access to data sources, construction of custom functions, and configuration of data pipelines, the lake-warehouse integration system provided in the first embodiment of the present disclosure includes: a data source access module 101, which accesses the target data source based on the requirements of the second object; a function construction module 102, which constructs the first custom function based on the processing requirements for the raw data information; and a data pipeline configuration module 103, which configures the target data pipeline based on the first custom function and processing requirements, enabling execution of the target data pipeline in the lake-warehouse integration system. Optionally, as shown in Figure 2, the data exploration layer 10 includes a data source access module 101, a function construction module 102, and a data pipeline configuration module 103. The data exploration layer 10 primarily includes data source access, ML-UDF development (i.e., the aforementioned function construction), and data pipeline configuration. In the data source access module 101, the system automatically connects to data sources such as MySQL (a relational database management system), Kafka (a high-throughput distributed publish-subscribe messaging system) message queues, and Push API (Application Programming Interface) endpoints for data synchronization and access. The function construction module 102 provides multimodal data management capabilities and allows custom function development using a programming language selected based on task requirements. Sample input is also provided to verify the correctness of the ML-UDF, and the data pipeline layer, based on stream computing capabilities, cleans and processes the data. In the data pipeline module 103, users can use DSL or SQL syntax to define pipeline tasks by themselves. For example, users can use SQL, JSON (JavaScript Object Notation) or YAML (YAML). ,Data pipelines can be defined using DSLs such as the Data Markup Language (DML), or by dragging corresponding operators using a drag-n-drop approach for visual data pipeline assembly. It should be noted that a large language model can be configured in the data source access module 101, the function construction module 102, and the data pipeline configuration module 103. This large language model enables intelligent data source access, data pipeline configuration, and user-defined function (UDF) generation. This configuration allows for rapid data source access, custom function construction, and data pipeline configuration, thereby improving data processing efficiency. To facilitate subsequent data applications and data sharing, the lake-warehouse integration system provided in the first embodiment of the present disclosure includes: a raw data layer 201 for storing raw data information imported from a target data source; a pipeline layer 202 for executing a target data pipeline to process the raw data information and obtain first data information, where the first data information is required by a second object; and a lake-warehouse integration layer 203 for storing the first data information. Optionally, as shown in Figure 3, the lake-warehouse integration 20 includes a raw data layer 201, a pipeline layer 202, and a lake-warehouse integration layer 203. The raw data layer 201, also known as the Operational Data Store (ODS) layer, is used for data access and synchronization. Data access can be implemented in both push and pull modes. Traditional data sources use a single stream computing engine for unified data entry into the lake, while also providing a Push Gateway component for standardized processing of records containing multimodal information. The pipeline layer 202, leveraging the capabilities of the Stream Compute DAG (Directed Acyclic Graph) and ML-UDF (the aforementioned custom functions), streams ODS layer data and performs operations such as data processing, data mining, data pre-computation, and data-derived metrics. The pipeline layer 202 includes custom functions capable of processing multimodal records, allowing users to use different programming languages ​​for data processing in different scenarios. Finally, with the ability to write data through data diversion, the streaming pipeline layer can simultaneously provide data to the DWD (data warehouse details, data warehouse layer) / DWS (data warehouse middle, data service layer) / ADS (application data service, application data layer) layers in the traditional lake warehouse architecture.The lake-warehouse integration layer 203 combines the capabilities of a lake and a warehouse, providing data support for database and big data engine analysis and data access interfaces (APIs). Users can query data using MySQL syntax or perform big data analysis using big data recorders such as Flink. Furthermore, the lake-warehouse integration layer 203 provides multimodal data management and query interfaces, allowing users to manage and query multimodal data produced by different data pipelines. Regarding storage, the lake-warehouse layer stores the following four types of data: processed and processed data (DWD), data corresponding to upper-layer application APIs (DWS), datasets created for specific application requirements (ADS), and multimodal data produced by the streaming pipeline layer. In summary, the lake-warehouse integration 203 can process and analyze various types of data, such as images, audio, and video, enabling more comprehensive processing of multimodal information and providing corresponding data services and applications. In the lake-warehouse integrated system provided in the first embodiment of the present disclosure, the data application layer 30 further includes: a data management module for managing first data information; a data query and analysis module for querying and analyzing the first data information to obtain data information required by the first object; and a data application module for processing and handling the first data information to meet task requirements. Optionally, as shown in FIG4 , the data application layer 30 further includes: a data management module 301, a data query and analysis module 302, and a data application module 303. The data management module 301 is used to manage the first data information, which includes vectors, videos, audio, images, files, web page information, etc. The data query and analysis module 302 supports data analysis and query using databases such as MySQL, Flink SQL, and Spark SQL, as well as big data technologies; and the data application module 303 is used to process and handle the first data information to meet task requirements. For example, one-click construction of data APL data reports, data visualization, large models, and machine learning is possible. In the lake-warehouse integrated system provided in the first embodiment of the present disclosure, the data exploration layer 10 also includes a metadata processing module. The metadata processing module is used to detect the first data information stored in the lake-warehouse integrated layer to obtain metadata information corresponding to the first data information. Optionally, as shown in Figure 5, the data exploration layer 10 also includes a metadata processing module 104. This module can periodically detect the lake-warehouse table structure and partition structure, and perform data and partition expiration operations.Users can also provide a file path to the metadata processing module 104, which retrieves all supported libraries and tables under that path and incorporates them into metadata management. It should be noted that metadata is information about data organization, data domains, and their relationships. Simply put, metadata is used to describe data. In an optional embodiment, the end-to-end flow chart of the lake-warehouse integrated system provided in the first embodiment of the present disclosure is shown in Figure 6. This flow primarily includes data insights (corresponding to the aforementioned data exploration module), building a streaming lake warehouse (corresponding to the aforementioned lake-warehouse integrated system), and data applications (corresponding to the aforementioned data application layer). The data insights module primarily includes data source access, ML-UDF development, and data pipelines. Building a streaming lake warehouse primarily includes the ODS layer, the data pipeline layer (corresponding to the aforementioned pipeline layer), the lake warehouse layer (corresponding to the aforementioned lake-warehouse integrated system), and the ODS (raw data storage) layer: responsible for connecting to data sources such as MySQL databases, Kafka message queues, and Push API endpoints for data synchronization and access. The ODS layer provides capabilities such as incremental streaming reads, TTL (Time To Live), and data replay. The data pipeline layer performs multimodal data processing and stores the resulting data in the DWD (Data Wide Surface Layer), DWS (Data Service Layer), and ADS (Application Data Layer). The lake-warehouse layer provides both lake (raw data storage) and warehouse (data processing) capabilities, and provides analytical interfaces for users to query and analyze data. The data application block primarily includes the application data platform (APL) multimodal data management, large model, and machine learning modules. In an optional embodiment, the aforementioned lake-warehouse integrated system can be used for cloud service provider access. Through the lake-warehouse integrated system, cloud service providers can integrate this data development and integration platform into their official websites. The scheduled scheduling function can be used to build a QA Bot to improve the accuracy of responses to self-service tickets. By organizing common questions and answers into a dataset and combining machine learning and natural language processing technologies, the QA Bot can automatically respond to user questions and provide accurate solutions. This will significantly improve user satisfaction and ticket processing efficiency. In an optional embodiment, the aforementioned Lake Warehouse integrated system can also be integrated with camera manufacturers. Simply using a standard internet-connected camera, the system can leverage cloud computing and AI capabilities to implement real-time fall detection and sound detection alerts. By streaming real-time video from the camera to the cloud, this solution can utilize AI algorithms to detect falls and send corresponding alerts.At the same time, abnormal sounds can also be identified through sound detection algorithms, and alarms can be issued in time to ensure the safety and convenience of users. In an optional embodiment, the above-mentioned Lake Warehouse Integrated System can also be used for children's smart watch access: based on real-time geographic location information, an abnormal activity range alarm can be realized. Through the positioning function on the watch, the child's activity range can be detected. If the child exceeds the preset safety range, an alarm notification will be sent to the guardian to ensure the child's safety. In an optional embodiment, the above-mentioned Lake Warehouse Integrated System can also be used for Call Center access: For call centers, different UDFs can be deployed according to different scenarios to achieve real-time monitoring and analysis of indicators such as the number of calls answered, average call duration, and satisfaction evaluation. Through the Lake Warehouse Integrated System, the call center can detect and analyze call data in real time, and generate real-time alarms and reports based on the set rules and indicators. This will help call centers monitor call quality and task performance, and improve the quality and efficiency of customer service. In an optional embodiment, the aforementioned integrated lake and warehouse system can also be used for autonomous driving integration: collecting and analyzing driver driving data, assessing driving behavior and personality traits, to provide a more intelligent and personalized driving experience and improve vehicle safety and performance. For autonomous vehicles, this solution can access the driver's driving data, analyze and evaluate it, and determine information such as the driver's personality and behavior. By collecting driving data, such as vehicle sensor data and driving behavior data, this solution can apply machine learning and data analysis algorithms to extract relevant driver characteristics and behavior patterns, and then analyze and evaluate their driving behavior, such as their driving style, concentration, and safety awareness. This information is of great significance for optimizing and improving autonomous driving systems. By understanding driver behavioral characteristics, autonomous driving systems can better adapt to driver needs and habits, improving the driving experience and safety. This solution also provides real-time driving behavior detection and alerting capabilities. By interacting with real-time data from vehicle sensors, it can detect abnormal driving behavior and issue timely alerts to prompt the driver to take appropriate measures. This helps improve driver safety awareness and driving behavior, reducing the risk of traffic accidents.In the lake-warehouse integrated system provided in the first embodiment of the present disclosure, a data exploration layer is used, wherein the data exploration layer is used to access the target data source, construct a first custom function and configure a target data pipeline, the target data source includes the original data information to be processed, and the target data pipeline calls the first custom function to process the original data information; the lake-warehouse integrated system, wherein the target data pipeline is executed in the lake-warehouse integrated system to process the original data information to obtain the first data information, and the lake-warehouse integrated system is also used to store the first data information; the data application layer, wherein the first object calls the first data information through the data application layer, solves the technical problem in the related art that the lake-warehouse integrated system requires the user to manually configure the data source, table structure, resource and other information before processing and storing the data, resulting in relatively low data processing efficiency. In this solution, the target data source is accessed through the data exploration layer, a first custom function is constructed, and a target data pipeline is configured. The target data pipeline is then executed in the lake-warehouse integration to process the original data information and obtain the first data information. This avoids the need for users to manually configure information such as the data source, table structure, and resources in order to process and store the data. The data exploration layer can quickly access the target data source, accurately construct the first custom function, and configure the corresponding target data pipeline, thereby achieving the effect of improving data processing efficiency. Example 2 According to an embodiment of the present disclosure, a data processing method based on the lake-warehouse integration system is also provided. Under the operating environment of the lake-warehouse integration system provided in the above-mentioned embodiment 1, the present disclosure provides a data processing method based on the lake-warehouse integration system as shown in Figure 7. Figure 7 is a flowchart of the data processing method based on the lake-warehouse integration system according to the second embodiment of the present disclosure. The method includes: Step S701, receiving the original data information to be processed from the target data source through the lake-warehouse integration in the lake-warehouse integration system. Optionally, when the target data source is connected, the lake-warehouse integration in the lake-warehouse integration system receives the original data information to be processed from the target data source. It should be noted that after obtaining the above-mentioned original data information, unified ODS storage will be performed, that is, the original data information will be directly stored in the original data layer in the lake-warehouse integration, providing consistent data incremental stream reading (incremental read) > incremental query (incremental query), data expiration (TTL), data partitioning, data replay and other capabilities for subsequent data pipelines.It should be noted that the above-mentioned methods for obtaining raw data include but are not limited to accessing a data source of database type, queue type, or log type, or using a push method, in which the task client pushes data to a specific endpoint of the lake-warehouse integrated system. Step S702: When there is no custom function for processing raw data information in the data exploration layer of the lake-warehouse integrated system, a first custom function is generated based on the processing requirements of the raw data information through the data exploration layer, and a target data pipeline is generated based on the processing requirements and the first custom function, wherein the target data pipeline calls the first custom function to process the raw data information. Optionally, after obtaining the raw data information, it is first necessary to determine whether there is a custom function for processing the raw data information. If the data exploration layer already has a custom function that can process the raw data information, the data management corresponding to the custom function is directly called to process the raw data. If the data exploration layer in the integrated lake-warehouse system does not have custom functions for processing raw data, a first custom function can be generated based on the raw data processing requirements. The data exploration layer can include a visualization interface to capture raw data processing requirements, such as data format conversion, data pre-calculation, or calculation of data-derived metrics. A corresponding first custom function (ML-UDF) is then generated based on the raw data processing requirements. It should be noted that the first custom function can generate multimodal data, such as using JavaScript and Chromium to crawl field links to obtain webpage information, using R to batch generate real-time statistical charts for statistical fields, or vectorizing text information to generate vector data. Furthermore, the first custom function offers cross-language support, allowing users to use different programming languages ​​to obtain corresponding custom functions based on task requirements. The visualization interface can also be used to capture data pipeline configuration requirements, and the target data pipeline is generated based on the processing requirements and the first custom function. By executing the target data pipeline, the first custom function can be called to quickly process the raw data information, thereby obtaining the data information required by the user. In step S703, the lake warehouse integration in the lake warehouse integration system processes the raw data information according to the target data pipeline to obtain the first data information. Optionally, the target data pipeline is executed in the lake warehouse integration in the lake warehouse integration system, and the first custom function is called by the target data pipeline to quickly process the raw data information, ultimately obtaining the data information required by the user.In summary, the above-mentioned custom functions can meet complex data processing requirements. Users can choose the most suitable programming language according to task requirements to process and process data, and can effectively reduce development time and workload. In order to improve the access efficiency of the data source, in the data processing method based on the lake warehouse integrated system provided in the second embodiment of the present disclosure, before receiving the original data information to be processed from the target data source through the lake warehouse integrated system, the method also includes: receiving the first information input by the first target object through the first front-end interface corresponding to the data source access module in the data exploration layer, and sending the first information to the data source access module through the first front-end interface, wherein the first information at least includes the identification information of the data source; determining the target data source based on the target language model and the identification information in the first information through the data source access module, and pushing the parameter information corresponding to the target data source to the first front-end interface; receiving the second information input by the first target object through the first front-end interface, and establishing a connection with the target data source based on the second information through the data source access module to receive the original data information to be processed from the target data source, wherein the second information at least includes the identity identification ID information of the first target object. Optionally, the Lake-Warehouse Integration system includes a data source access module, a function construction module, and a data pipeline configuration module. The data source access module is primarily used for data source access. To improve data source access efficiency, a target language model is provided within the data source access module, enabling rapid data source access through AI assistance. As shown in Figure 8, a first message input by a first target subject is received within a first front-end interface corresponding to the data access module. It should be noted that the first message includes at least the identification information of the data source. For example, the first message may be a request to access a cloud database in region xx. Based on the identification information in the first message, the target language model identifies the target data source to be accessed and pushes the target data source's data information (i.e., the aforementioned parameter information) to the first front-end interface corresponding to the data access module. The first front-end interface then receives a second message. It should be noted that the second message includes at least the ID information of the first target subject, such as a username and password. Finally, the data access module establishes a connection with the target data source to receive the raw data information to be processed from the target data source. The aforementioned target language model can effectively reduce resource and network configuration processes and improve data source access efficiency.In order to effectively improve the development efficiency of custom functions, in the data processing method based on the lake-warehouse integrated system provided in the second embodiment of the present disclosure, the third information input by the first target object is received through the second front-end interface corresponding to the function construction module in the data exploration layer, and the third information is sent to the function construction module through the second front-end interface, wherein the third information is the description information of the processing requirements; the function construction module determines whether the target tool needs to be called based on the target language model to obtain a judgment result; when the judgment result indicates that the target tool needs to be called, the function construction module processes the third information based on the target language model and the target tool to obtain a first custom function; and the function construction module pushes the first custom function to the second front-end interface. Optionally, the function construction module in the data exploration layer is primarily used to generate the aforementioned first custom function. A natural language model is provided within the function construction module, leveraging AI to rapidly generate custom functions. Users can enter their data processing requirements for the data source, namely the aforementioned third information, in the second front-end interface corresponding to the function construction module. The target language model then determines whether a target tool, such as FetchUdfTemplate() (the UDF template tool), needs to be invoked. If so, the target language model processes the third information based on the target language model and the target tool, automatically generating the aforementioned first custom function. After the first custom function is generated, it is pushed to the second front-end interface, where the user can also adjust it. It should be noted that if a custom function already exists that meets the user's data processing requirements, it can be directly invoked without executing the aforementioned custom function generation process. It should be noted that during the custom function generation process, users can design different programming languages ​​based on actual needs and configure the custom function's multimodal information processing capabilities. It should be noted that the first custom function described above can be packaged as a native-image (a technology that generates natively executable text in a Java virtual machine) and deployed similarly to a serverless function. It should be noted that in addition to input and output in standard record formats, the first custom function described above also produces multimodal data. For example, it uses JavaScript Chromium to crawl field links to obtain web page information, uses R to batch generate real-time statistical charts for statistical fields, and vectorizes text information to generate vector data.The first custom function will upload the generated multimodal data to the multimodal information path and write it in the form of a side-output stream. In summary, by using multi-programming language support and ML-UDF development functions, more flexible data processing capabilities are provided. Users can choose the most suitable programming language according to task requirements to process and process data, thereby improving user experience, as well as data integration and development efficiency. In order to ensure the availability of the first custom function, in the data processing method based on the lake warehouse integrated system provided in the second embodiment of the present disclosure, after obtaining the first custom function, the method also includes: obtaining a preset custom function specification through a function construction module; testing the first custom function according to the custom function specification through the function construction module to obtain a test result; when the test result indicates that there is no abnormality in the first custom function, compiling and deploying the first custom function through the function construction module so that the target data pipeline can call the first custom function for data processing. Optionally, a preset custom function specification is provided in the function construction module. After generating the first custom function, the function construction module invokes the custom function specification to test the first custom function to determine whether there are any issues with the first custom function. If the test result indicates that the first custom function is not abnormal, the function construction module compiles and deploys the first custom function so that the target data pipeline can call the first custom function for data processing. It should be noted that compiling and deploying the first custom function in the function construction module primarily enables the first custom function to be called and executed by the data pipeline at the pipeline layer. By testing the first custom function, the usability of the first custom function can be effectively determined, avoiding issues such as inaccurate data processing during subsequent data processing. In the prior art, the lake-warehouse integration process prevents users from intuitively understanding the processing of the custom function and whether the processed data structure meets their requirements during the initial development and compilation process, which seriously affects development efficiency and user experience.To address the aforementioned issues, in a data processing method based on a lake-warehouse integrated system, provided in a second embodiment of the present disclosure, after the data exploration layer in the lake-warehouse integrated system generates a first custom function based on processing requirements for raw data information, the method further includes: invoking a node-based workflow automation tool via a function construction module in the data exploration layer to compile the first custom function to obtain a compiled first custom function; constructing a workflow based on the compiled first custom function via the function construction module, wherein the workflow verifies whether the first custom function meets the processing requirements; and executing the workflow via the function construction module to obtain an execution result, and verifying whether the first custom function meets the processing requirements based on the execution result. Optionally, after obtaining the first custom function via the function construction module, the node-based workflow automation tool can be invoked to compile the first custom function to obtain a compiled first custom function. Then, in the front-end interface corresponding to the function construction module, a workflow can be constructed based on the compiled first custom function. For example, the workflow shown in FIG9 includes: a data pipeline deployment node, a data source node, a first custom function node, and an online analysis node. The workflow is executed to determine whether the first custom function meets the processing requirements. It should be noted that the nodes in the workflow described above are pluggable, allowing users to configure the execution order of each workflow node based on their actual needs. It should be noted that during execution, the workflow described above will perform real data processing based on small batches of data obtained from the data source by the custom function, obtaining data processing results to accurately determine whether the custom function meets the required data processing requirements. In an optional embodiment, the processing process of the first custom function is shown in Figure 10. The custom function is generated, and then the source code of the custom function is obtained using CodeGen (a compilation tool). This is then compiled using Flink to generate Java code, facilitating the subsequent invocation of the custom function in a real data pipeline. Simultaneously, the source code of the custom function is compiled using a node-based workflow automation tool to construct a workflow to verify the data processing capabilities of the custom function. Accurately verifying the data processing capabilities of the custom function through the workflow described above achieves a more efficient and convenient data development process, enabling users to quickly build data applications and services.To improve the configuration efficiency of the target data pipeline, in the data processing method based on the lake-warehouse integrated system provided in the second embodiment of the present disclosure, generating the target data pipeline based on processing requirements and a first custom function includes: determining a data pipeline template based on the target language model and processing requirements by a data pipeline configuration module in the data exploration layer; configuring the data pipeline based on the data pipeline template and the first custom function by the data pipeline configuration module to obtain configuration information; and generating the target data pipeline based on the configuration information by the data pipeline configuration module. Optionally, the data pipeline configuration module is provided with a target language model, and the data pipeline template can be determined based on the target language model and processing requirements. The data pipeline can then be configured using the data pipeline template and the first custom function to obtain configuration information (e.g., JSON / YAML configuration information for the data pipeline), ultimately resulting in the target data pipeline. It should be noted that the target data pipeline can include multiple custom functions, and the target data pipeline can process data according to a directed acyclic graph. To ensure data writing accuracy, in the data processing method based on the lake-warehouse integrated system provided in Example 2 of the present disclosure, after receiving the raw data information to be processed from the target data source via the lake-warehouse integrated system, the method further includes: determining the data source type of the target data source via the lake-warehouse integrated system; determining a write operator based on the data source type via the lake-warehouse integrated system; and writing the raw data information into the raw data layer of the lake-warehouse integrated system via the lake-warehouse integrated system based on the write operator. Optionally, after receiving the raw data information to be processed from the target data source via the lake-warehouse integrated system, a write operator can be determined based on the data source type of the target data source, and then the raw data information is written to the ODS layer (i.e., the raw data layer described above) based on the corresponding write operator. For example, MySQL data sources are written using Flink CDC + Debezium; Apache Kafka message queue data is written using the Flink Kafka Connector; and log-type data is written using the Flink Log Connector. Setting different write operators for different data types can effectively improve data writing accuracy.In an optional embodiment, the data processing process is shown in Figure 11. Various data sources, such as MySQL, Apache Kafka message queue data, and log data, are accessed. Raw data from these data sources is written to the ODS layer, providing consistent capabilities for subsequent pipelines, such as incremental reads, incremental queries, time-to-live (TTL) settings, data partitioning, and data replay. The pipeline layer streams ODS data for operations such as data processing, data mining, data pre-calculation, and data-derived metrics. The lake warehouse layer stores processed and processed data (DWD), data corresponding to upper-layer application APIs (DWS), datasets created for specific application requirements (ADS), and multi-modal data generated by the streaming pipeline layer. In an optional embodiment, a schematic diagram illustrating the use of AI (i.e., the target language model described above) to intelligently access data sources, configure data pipelines, and generate user-defined functions (UDFs) is shown in Figure 12. Users ask questions in the front-end interface of the corresponding module (i.e., modules such as accessing data sources, configuring data pipelines, and generating UDFs). Based on the user input, the system determines whether a tool is required. If so, the tool is invoked to output the structured data (e.g., UDFs and configuration information) required by the user. The system then determines whether the output structured data meets the output specifications. If so, the output results are pushed to the front-end interface for user confirmation. Using AI to intelligently access data sources, configure data pipelines, and generate UDFs can effectively improve the efficiency of subsequent data processing. In an optional embodiment, the aforementioned custom function can also implement seamless link upgrades. From the perspective of a single custom function, the upgrade process for a custom function may include the following steps: Upon receiving an operator event (i.e., an upgrade event), the custom function begins caching original records. A global snapshot is taken through an operator-triggered checkpoint. Upon completion of the snapshot, the custom function switches to a new ML-UDF and replays the unprocessed records.From the perspective of a directed acyclic graph, the custom function upgrade process can include the following steps: When the upstream custom function upgrade is complete, information about the ML-UDF switch is transmitted downstream; downstream operators upgrade the ML-UDF version only after receiving watermarks from all upstream custom functions; otherwise, records are cached and processing continues using the old ML-UDF until all watermarks expire, switching to the new ML-UDF and replaying unprocessed records. This custom function upgrade process enables seamless link upgrades, improving the user experience. In the data processing method based on the lake-warehouse integrated system provided in Example 2 of the present disclosure, after the lake-warehouse integrated system processes raw data information based on the target data pipeline to obtain first data information, the method further includes: receiving a data query request initiated by a second target object via a third front-end interface corresponding to the data application layer in the lake-warehouse integrated system, and sending the data query request to the data application layer via the third front-end interface; obtaining, via the data application layer, second data information corresponding to the data query request from the lake-warehouse integrated layer of the lake-warehouse integrated system based on the data query request; and pushing the second data information to the third front-end interface via the data application layer. Optionally, after the data processing and data storage are completed, the user (i.e., the second target object mentioned above) can initiate a data query request through the third front-end interface corresponding to the data application layer in the lake warehouse integrated system. The third front-end interface sends the data query request to the data application layer. Then, the data application layer obtains the second data information corresponding to the data query request from the lake warehouse integrated layer according to the data query request, and pushes the queried second data information to the third front-end interface, so that the user can perform the corresponding task according to the queried data. In the data processing method based on the lake warehouse integrated system provided in the second embodiment of the present disclosure, after the original data information is processed according to the target data pipeline by the lake warehouse integrated system in the lake warehouse integrated system to obtain the first data information, the method also includes: constructing a training sample set according to the first data information through the data application layer; training the neural network model under the target scenario corresponding to the first data information according to the training sample set through the data application layer to obtain the target neural model.Optionally, after data processing and data storage, the data information stored in the lake-warehouse integrated layer (e.g., the first data information described above) can also be used to construct a training sample set. This training sample set can then be used to train a neural network model for the target scenario corresponding to the first data information. For example, in the call center field, a training sample set can be constructed based on data such as the number of calls answered and average call duration. Model training can then be performed to obtain a target neural model, which can then be used to dynamically adjust call resources. For example, in the autonomous driving field, a training sample set can be constructed based on driver driving data, evaluating driving behavior and personality traits. Model training can then be performed to obtain a target neural model, which can then be used to provide a more intelligent and personalized driving experience. In summary, the data processing method based on the lake-warehouse integrated system provided by the embodiments of the present disclosure meets complex data processing needs through multi-programming language support and ML-UDF development. It also effectively improves development efficiency and convenience through a visual interface, intelligent configuration in data exploration, automated task scheduling, and pipeline configuration. It also provides general SQL and multi-modal semantic support at the data application level, supports large data APL models and machine learning, multi-modal data management, query, and analysis, and enables more efficient and convenient data development. In the data processing method based on the lake-warehouse integrated system provided in the second embodiment of the present disclosure, the lake-warehouse integrated system receives the original data information to be processed from the target data source; the data exploration layer in the lake-warehouse integrated system generates a first custom function based on the processing requirements of the original data information, and generates a target data pipeline based on the processing requirements and the first custom function, wherein the target data pipeline calls the first custom function to process the original data information; the lake-warehouse integrated system processes the original data information based on the target data pipeline to obtain the first data information, which solves the technical problem in the related technology that the lake-warehouse integrated system requires the user to manually configure the data source, table structure, resource and other information before processing and storing the data, resulting in relatively low data processing efficiency.In this solution, the data exploration layer accesses the target data source, constructs a first custom function, and configures the target data pipeline. The target data pipeline is then executed in the lake-warehouse integration to process the raw data information and obtain the first data information. This eliminates the need for users to manually configure information such as data sources, table structures, and resources for data processing and storage. The data exploration layer allows for rapid access to the target data source, accurate construction of the first custom function, and configuration of the corresponding target data pipeline, thereby improving data processing efficiency. It should be noted that the aforementioned method embodiments are presented as a series of actions for simplicity. However, those skilled in the art should understand that this disclosure is not limited by the order of the actions described, as certain steps may be performed in a different order or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in this specification are preferred embodiments, and the actions and modules involved are not necessarily required by this disclosure. Through the above description of the embodiments, those skilled in the art will clearly understand that the methods according to the aforementioned embodiments can be implemented using software and a necessary general-purpose hardware platform. Hardware can also be used, but in many cases, the former is a more preferred implementation. Based on this understanding, the technical solution of this disclosure, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (e.g., ROM / RAM, a magnetic disk, or an optical disk) and includes instructions for enabling a terminal device (such as a mobile phone, a computer, a server, or a network device) to execute the methods described in the various embodiments of this disclosure. It should be noted that the preferred implementation schemes involved in the above-mentioned embodiments of this disclosure are the same as the solution, application scenarios, and implementation processes provided in Example 2, but are not limited to the solution provided in Example 2.Example 3 According to an embodiment of the present disclosure, an out-of-band network construction device for implementing the above-mentioned out-of-band network construction method is also provided. As shown in Figure 13, the device includes: a first receiving unit 1301, a generating unit 1302 and a processing unit 1303. The first receiving unit 1301 is used to receive the original data information to be processed from the target data source through the lake warehouse integration in the lake warehouse integration system; the generating unit 1302 is used to generate the first custom function based on the processing requirements of the original data information through the data exploration layer in the lake warehouse integration system when there is no custom function for processing the original data information, and generate a target data pipeline based on the processing requirements and the first custom function, wherein the target data pipeline calls the first custom function to process the original data information; the processing unit 1303 is used to process the original data information according to the target data pipeline through the lake warehouse integration in the lake warehouse integration system to obtain first data information. In the data processing device based on the lake-warehouse integrated system provided in the third embodiment of the present disclosure, the first receiving unit 1301 receives the original data information to be processed from the target data source through the lake-warehouse integrated system in the lake-warehouse integrated system; the generation unit 1302 generates the first custom function based on the processing requirements of the original data information through the data exploration layer when there is no custom function for processing the original data information in the data exploration layer of the lake-warehouse integrated system, and generates a target data pipeline based on the processing requirements and the first custom function, wherein the target data pipeline calls the first custom function to process the original data information; the processing unit 1303 processes the original data information according to the target data pipeline through the lake-warehouse integrated system in the lake-warehouse integrated system to obtain the first data information, which solves the technical problem in the related art that the lake-warehouse integrated system requires the user to manually configure the data source, table structure, resource and other information before processing and storing the data, resulting in relatively low data processing efficiency. In this solution, the target data source is accessed through the data exploration layer, the first custom function is built, and the target data pipeline is configured. Then, the target data pipeline is executed in the lake-warehouse integration to process the original data information and obtain the first data information. This avoids the need for users to manually configure information such as data sources, table structures, and resources for data processing and storage. The data exploration layer can quickly access the target data source, accurately build the first custom function, and configure the corresponding target data pipeline, thereby achieving the effect of improving data processing efficiency.Optionally, in the data processing device based on the lake-warehouse integrated system provided in Example 3 of the present disclosure, the device also includes: a second receiving unit, which is used to receive the first information input by the first target object through the first front-end interface corresponding to the data source access module in the data exploration layer before receiving the original data information to be processed from the target data source through the lake-warehouse integrated system, and send the first information to the data source access module through the first front-end interface, wherein the first information at least includes the identification information of the data source; a first determination unit, which is used to determine the target data source based on the target language model and the identification information in the first information through the data source access module, and push the parameter information corresponding to the target data source to the first front-end interface; a third receiving unit, which is used to receive the second information input by the first target object through the first front-end interface, and establish a connection with the target data source based on the second information through the data source access module to receive the original data information to be processed from the target data source, wherein the second information at least includes the identity identification ID information of the first target object. Optionally, in the data processing device based on the lake-warehouse integrated system provided in Example 3 of the present disclosure, the generation unit includes: a receiving module, which is used to receive the third information input by the first target object through the second front-end interface corresponding to the function construction module in the data exploration layer, and send the third information to the function construction module through the second front-end interface, wherein the third information is the description information of the processing requirement; a judgment module, which is used to judge whether it is necessary to call the target tool based on the target language model through the function construction module to obtain a judgment result; a processing module, which is used to process the third information based on the target language model and the target tool through the function construction module when the judgment result indicates that the target tool needs to be called, to obtain the first custom function; a pushing module, which is used to push the first custom function to the second front-end interface through the function construction module.Optionally, in the data processing device based on the lake-warehouse integrated system provided in the third embodiment of the present disclosure, the device also includes: a first acquisition unit, used to obtain a preset custom function specification through the function construction module after obtaining the first custom function; a verification unit, used to verify the first custom function according to the custom function specification through the function construction module to obtain a verification result; a first compilation unit, used to compile and deploy the first custom function through the function construction module when the verification result indicates that there is no abnormality in the first custom function, so that the target data pipeline can call the first custom function for data processing. Optionally, in the data processing device based on the lake-warehouse integrated system provided in Example 3 of the present disclosure, the device also includes: a second compilation unit, which is used to generate the first custom function based on the processing requirements of the original data information through the data exploration layer in the lake-warehouse integrated system, and then call the node-based workflow automation tool through the function construction module in the data exploration layer to compile the first custom function to obtain the compiled first custom function; a first construction unit, which is used to construct a workflow based on the compiled first custom function through the function construction module, wherein the workflow is used to verify whether the first custom function meets the processing requirements; an execution unit, which is used to execute the workflow through the function construction module, obtain an execution result, and verify whether the first custom function meets the processing requirements based on the execution result. Optionally, in the data processing device based on the lake-warehouse integrated system provided in Example 3 of the present disclosure, the generation unit includes: a determination module, which is used to determine the data pipeline template based on the target language model and the processing requirements through the data pipeline configuration module in the data exploration layer; a configuration module, which is used to configure the data pipeline according to the data pipeline template and the first custom function through the data pipeline configuration module to obtain configuration information; and a generation module, which is used to generate the target data pipeline according to the configuration information through the data pipeline configuration module.Optionally, in the data processing device based on the lake-warehouse integrated system provided in the third embodiment of the present disclosure, the device also includes: a second determination unit, which is used to determine the data source type of the target data source through the lake-warehouse integrated system after receiving the original data information to be processed from the target data source through the lake-warehouse integrated system; a third determination unit, which is used to determine the write operator through the lake-warehouse integrated system based on the data source type; and a writing unit, which is used to write the original data information into the original data layer of the lake-warehouse integrated system through the lake-warehouse integrated system based on the write operator. Optionally, in the data processing device based on the lake-warehouse integrated system provided in the third embodiment of the present disclosure, the device also includes: a fourth receiving unit, which is used to process the original data information according to the target data pipeline through the lake-warehouse integrated system to obtain the first data information, and then receive the data query request initiated by the second target object through the third front-end interface corresponding to the data application layer in the lake-warehouse integrated system, and send the data query request to the data application layer through the third front-end interface; a second acquisition unit, which is used to obtain the second data information corresponding to the data query request from the lake-warehouse integrated layer of the lake-warehouse integrated system through the data application layer according to the data query request; and a pushing unit, which is used to push the second data information to the third front-end interface through the data application layer. Optionally, in the data processing device based on the lake warehouse integrated system provided in the third embodiment of the present disclosure, the device further includes: a second construction unit for processing the original data information according to the target data pipeline through the lake warehouse integrated system to obtain the first data information, and then constructing a training sample set according to the first data information through the data application layer; a training unit for training the neural network model in the target scenario corresponding to the first data information according to the training sample set through the data application layer to obtain a target neural model. It should be noted here that the above-mentioned first receiving unit 1301, generation unit 1302 and processing unit 1303 correspond to steps S701 to S703 in the second embodiment, and the examples and application scenarios implemented by the three units and the corresponding steps are the same, but are not limited to the contents disclosed in the first embodiment. It should be noted that the preferred implementation scheme involved in the above embodiments of the present disclosure is the same as the scheme, application scenario and implementation process provided in the second embodiment, but is not limited to the scheme provided in the second embodiment. Example 4 The embodiment of the present disclosure can provide an electronic device, as shown in Figure 14, which can be any electronic device in a group of electronic devices.Optionally, in this embodiment, the above-mentioned electronic device can also be replaced with a terminal device such as a mobile terminal. Optionally, in this embodiment, the above-mentioned electronic device can be located in at least one network device among multiple network devices of a computer network. In this embodiment, the above-mentioned electronic device can execute the program code of the following steps in the data processing method based on the lake-warehouse integrated system: receiving the original data information to be processed from the target data source through the lake-warehouse integrated system in the lake-warehouse integrated system; in the case that the data exploration layer in the lake-warehouse integrated system does not have a custom function for processing the original data information, generating the first custom function based on the processing requirements of the original data information through the data exploration layer, and generating a target data pipeline based on the processing requirements and the first custom function, wherein the target data pipeline calls the first custom function to process the original data information; processing the original data information according to the target data pipeline through the lake-warehouse integrated system in the lake-warehouse integrated system to obtain the first data information. The above-mentioned electronic device can execute the program code of the following steps in the data processing method based on the lake-warehouse integrated system: Before receiving the original data information to be processed from the target data source through the lake-warehouse integrated system, the method also includes: receiving the first information input by the first target object through the first front-end interface corresponding to the data source access module in the data exploration layer, and sending the first information to the data source access module through the first front-end interface, wherein the first information at least includes the identification information of the data source; determining the target data source based on the target language model and the identification information in the first information through the data source access module, and pushing the parameter information corresponding to the target data source to the first front-end interface; receiving the second information input by the first target object through the first front-end interface, and establishing a connection with the target data source based on the second information through the data source access module to receive the original data information to be processed from the target data source, wherein the second information at least includes the ID information of the first target object.The above-mentioned electronic device can execute the program code of the following steps in the data processing method based on the lake-warehouse integrated system: generating the first custom function based on the processing requirements of the original data information through the data exploration layer in the lake-warehouse integrated system includes: receiving the third information input by the first target object through the second front-end interface corresponding to the function construction module in the data exploration layer, and sending the third information to the function construction module through the second front-end interface, wherein the third information is the description information of the processing requirements; judging whether it is necessary to call the target tool based on the target language model through the function construction module to obtain a judgment result; when the judgment result indicates that the target tool needs to be called, processing the third information based on the target language model and the target tool through the function construction module to obtain the first custom function; pushing the first custom function to the second front-end interface through the function construction module. The above-mentioned electronic device can execute the program code of the following steps in the data processing method based on the lake-warehouse integrated system: After obtaining the first custom function, the method also includes: obtaining a preset custom function specification through the function construction module; testing the first custom function according to the custom function specification through the function construction module to obtain a test result; when the test result indicates that there is no abnormality in the first custom function, compile and deploy the first custom function through the function construction module, so that the target data pipeline can call the first custom function for data processing. The above-mentioned electronic device can execute the program code of the following steps in the data processing method based on the lake-warehouse integrated system: After the data exploration layer in the lake-warehouse integrated system generates the first custom function based on the processing requirements of the original data information, the method also includes: calling the node-based workflow automation tool through the function construction module in the data exploration layer to compile the first custom function to obtain the compiled first custom function; constructing a workflow based on the compiled first custom function through the function construction module, wherein the workflow is used to verify whether the first custom function meets the processing requirements; executing the workflow through the function construction module to obtain the execution result, and verifying whether the first custom function meets the processing requirements based on the execution result.The above-mentioned electronic device can execute the program code of the following steps in the data processing method based on the lake-warehouse integrated system: Generating the target data pipeline according to the processing requirements and the first custom function includes: determining the data pipeline template based on the target language model and the processing requirements through the data pipeline configuration module in the data exploration layer; configuring the data pipeline according to the data pipeline template and the first custom function through the data pipeline configuration module to obtain configuration information; generating the target data pipeline according to the configuration information through the data pipeline configuration module. The above-mentioned electronic device can execute the program code of the following steps in the data processing method based on the lake-warehouse integrated system: After receiving the raw data information to be processed from the target data source through the lake-warehouse integrated system in the lake-warehouse integrated system, the method also includes: determining the data source type of the target data source through the lake-warehouse integrated system; determining the write operator through the lake-warehouse integrated system based on the data source type; and writing the raw data information into the raw data layer of the lake-warehouse integrated system through the lake-warehouse integrated system based on the write operator. The above-mentioned electronic device can execute the program code of the following steps in the data processing method based on the lake warehouse integrated system: after the lake warehouse integrated system processes the original data information according to the target data pipeline to obtain the first data information, the method further includes: receiving a data query request initiated by a second target object through a third front-end interface corresponding to the data application layer in the lake warehouse integrated system, and sending the data query request to the data application layer through the third front-end interface; obtaining the second data information corresponding to the data query request from the lake warehouse integrated layer of the lake warehouse integrated system through the data application layer according to the data query request; pushing the second data information to the third front-end interface through the data application layer. The above-mentioned electronic device can execute the program code of the following steps in the data processing method based on the lake warehouse integrated system: after the lake warehouse integrated system processes the original data information according to the target data pipeline to obtain the first data information, the method further includes: constructing a training sample set based on the first data information through the data application layer; training the neural network model under the target scenario corresponding to the first data information through the data application layer based on the training sample set to obtain a target neural model. Optionally, FIG8 is a structural block diagram of an electronic device according to an embodiment of the present disclosure. As shown in FIG8 , the electronic device 10 may include: one or more (only one is shown in FIG8 ) processors, and a memory.The memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the out-of-band network construction method and apparatus in the embodiments of the present disclosure. The processor executes the software programs and modules stored in the memory to execute various functional applications and data processing, thereby implementing the above-mentioned out-of-band network construction method. The memory can include high-speed random access memory (RAM) and non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory can further include memory located remotely from the processor, which can be connected to the terminal 100 via a network. Examples of such networks include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof. The processor can call the information and application stored in the memory through the transmission device to perform the following steps: receiving the original data information to be processed from the target data source through the lake warehouse integrated system; when there is no custom function for processing the original data information in the data exploration layer of the lake warehouse integrated system, generating the first custom function based on the processing requirements of the original data information through the data exploration layer, and generating a target data pipeline based on the processing requirements and the first custom function, wherein the target data pipeline calls the first custom function to process the original data information; processing the original data information according to the target data pipeline through the lake warehouse integrated system in the lake warehouse integrated system to obtain the first data information. The above-mentioned processor can execute the program code of the following steps in the data processing method based on the lake-warehouse integrated system: Before receiving the original data information to be processed from the target data source through the lake-warehouse integrated system, the method also includes: receiving the first information input by the first target object through the first front-end interface corresponding to the data source access module in the data exploration layer, and sending the first information to the data source access module through the first front-end interface, wherein the first information at least includes the identification information of the data source; determining the target data source based on the target language model and the identification information in the first information through the data source access module, and pushing the parameter information corresponding to the target data source to the first front-end interface; receiving the second information input by the first target object through the first front-end interface, and establishing a connection with the target data source based on the second information through the data source access module to receive the original data information to be processed from the target data source, wherein the second information at least includes the ID information of the first target object.The above-mentioned processor can execute the program code of the following steps in the data processing method based on the lake-warehouse integrated system: Generating the first custom function based on the processing requirements of the original data information through the data exploration layer in the lake-warehouse integrated system includes: receiving the third information input by the first target object through the second front-end interface corresponding to the function construction module in the data exploration layer, and sending the third information to the function construction module through the second front-end interface, wherein the third information is the description information of the processing requirements; judging whether it is necessary to call the target tool based on the target language model through the function construction module to obtain a judgment result; when the judgment result indicates that the target tool needs to be called, processing the third information based on the target language model and the target tool through the function construction module to obtain the first custom function; pushing the first custom function to the second front-end interface through the function construction module. The above-mentioned processor can execute the program code of the following steps in the data processing method based on the lake-warehouse integrated system: After obtaining the first custom function, the method also includes: obtaining a preset custom function specification through the function construction module; testing the first custom function according to the custom function specification through the function construction module to obtain a test result; when the test result indicates that there is no abnormality in the first custom function, compile and deploy the first custom function through the function construction module, so that the target data pipeline can call the first custom function for data processing. The above-mentioned processor can execute the program code of the following steps in the data processing method based on the lake-warehouse integrated system: After the data exploration layer in the lake-warehouse integrated system generates the first custom function based on the processing requirements of the original data information, the method also includes: calling the node-based workflow automation tool through the function construction module in the data exploration layer to compile the first custom function to obtain the compiled first custom function; constructing a workflow based on the compiled first custom function through the function construction module, wherein the workflow is used to verify whether the first custom function meets the processing requirements; executing the workflow through the function construction module to obtain the execution result, and verifying whether the first custom function meets the processing requirements based on the execution result.The above-mentioned processor can execute the program code of the following steps in the data processing method based on the lake-warehouse integrated system: Generating the target data pipeline according to the processing requirements and the first custom function includes: determining the data pipeline template based on the target language model and the processing requirements through the data pipeline configuration module in the data exploration layer; configuring the data pipeline according to the data pipeline template and the first custom function through the data pipeline configuration module to obtain configuration information; generating the target data pipeline according to the configuration information through the data pipeline configuration module. The above-mentioned processor can execute the program code of the following steps in the data processing method based on the lake-warehouse integrated system: After receiving the original data information to be processed from the target data source through the lake-warehouse integrated system in the lake-warehouse integrated system, the method also includes: determining the data source type of the target data source through the lake-warehouse integrated system; determining the write operator through the lake-warehouse integrated system based on the data source type; writing the original data information into the original data layer of the lake-warehouse integrated system through the lake-warehouse integrated system based on the write operator. The above-mentioned processor can execute the program code of the following steps in the data processing method based on the lake warehouse integrated system: After the lake warehouse integrated system in the lake warehouse integrated system processes the original data information according to the target data pipeline to obtain the first data information, the method also includes: receiving the data query request initiated by the second target object through the third front-end interface corresponding to the data application layer in the lake warehouse integrated system, and sending the data query request to the data application layer through the third front-end interface; obtaining the second data information corresponding to the data query request from the lake warehouse integrated layer of the lake warehouse integrated system through the data application layer according to the data query request; pushing the second data information to the third front-end interface through the data application layer. The above-mentioned processor can execute the program code of the following steps in the data processing method based on the lake warehouse integrated system: After the lake warehouse integrated system in the lake warehouse integrated system processes the original data information according to the target data pipeline to obtain the first data information, the method also includes: constructing a training sample set based on the first data information through the data application layer; training the neural network model under the target scenario corresponding to the first data information through the data application layer based on the training sample set to obtain a target neural model.Those skilled in the art will appreciate that the structure shown in FIG14 is merely illustrative, and the electronic device may also be a terminal device such as a smartphone (such as an Android phone, an iOS phone, etc.), a tablet computer, a PDA, or a mobile internet device (MID), PAD, etc. FIG14 does not limit the structure of the above-mentioned electronic devices. For example, the electronic device may include more or fewer components (such as a network interface, a display device, etc.) than those shown in FIG14 , or have a configuration different from that shown in FIG14 . Those skilled in the art will appreciate that all or part of the steps in the various methods of the above-mentioned embodiments can be completed by a program instructing the hardware associated with the terminal device. The program may be stored in a computer-readable storage medium, which may include a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc. Example 5 The embodiments of the present disclosure also provide a storage medium. Optionally, in this embodiment, the storage medium may be used to store program code executed by the out-of-band network construction method provided in the first embodiment and program code executed by the out-of-band network-based out-of-band processing method provided in the second embodiment. Optionally, in this embodiment, the storage medium may be located in any computer terminal in a computer terminal group in a computer network, or in any mobile terminal in a mobile terminal group. Optionally, in this embodiment, the storage medium is configured to store program code for executing the following steps: receiving raw data information to be processed from a target data source via the lake warehouse integration in the lake warehouse integration system; generating a first custom function based on processing requirements for the raw data information via the data exploration layer in the lake warehouse integration system, if no custom function exists for processing the raw data information, and generating a target data pipeline based on the processing requirements and the first custom function, wherein the target data pipeline calls the first custom function to process the raw data information; and processing the raw data information via the lake warehouse integration in the lake warehouse integration system according to the target data pipeline to obtain first data information.The above-mentioned storage medium is configured to store program code for executing the following steps: Before receiving the original data information to be processed from the target data source through the lake-warehouse integrated system, the method also includes: receiving the first information input by the first target object through the first front-end interface corresponding to the data source access module in the data exploration layer, and sending the first information to the data source access module through the first front-end interface, wherein the first information at least includes the identification information of the data source; determining the target data source based on the target language model and the identification information in the first information through the data source access module, and pushing the parameter information corresponding to the target data source to the first front-end interface; receiving the second information input by the first target object through the first front-end interface, and establishing a connection with the target data source based on the second information through the data source access module to receive the original data information to be processed from the target data source, wherein the second information at least includes the ID information of the first target object. The above-mentioned storage medium is configured to store program codes for executing the following steps: generating the first custom function based on the processing requirements of the original data information through the data exploration layer in the lake warehouse integrated system includes: receiving the third information input by the first target object through the second front-end interface corresponding to the function construction module in the data exploration layer, and sending the third information to the function construction module through the second front-end interface, wherein the third information is the description information of the processing requirements; judging whether it is necessary to call the target tool based on the target language model through the function construction module to obtain a judgment result; when the judgment result indicates that the target tool needs to be called, processing the third information based on the target language model and the target tool through the function construction module to obtain the first custom function; pushing the first custom function to the second front-end interface through the function construction module. The storage medium is configured to store program code for executing the following steps: After obtaining the first custom function, the method further includes: obtaining a preset custom function specification through the function construction module; testing the first custom function according to the custom function specification through the function construction module to obtain a test result; and if the test result indicates that there is no abnormality in the first custom function, compiling and deploying the first custom function through the function construction module so that the target data pipeline can call the first custom function for data processing.The storage medium is configured to store program code for performing the following steps: After the data exploration layer in the lake-warehouse integrated system generates the first custom function based on the processing requirements for the raw data information, the method further includes: calling a node-based workflow automation tool to compile the first custom function through a function construction module in the data exploration layer to obtain a compiled first custom function; constructing a workflow based on the compiled first custom function through the function construction module, wherein the workflow is used to verify whether the first custom function meets the processing requirements; executing the workflow through the function construction module to obtain an execution result, and verifying whether the first custom function meets the processing requirements based on the execution result. The storage medium is configured to store program code for performing the following steps: generating a target data pipeline based on the processing requirements and the first custom function includes: determining a data pipeline template based on a target language model and the processing requirements through a data pipeline configuration module in the data exploration layer; configuring the data pipeline based on the data pipeline template and the first custom function through the data pipeline configuration module to obtain configuration information; and generating the target data pipeline through the data pipeline configuration module based on the configuration information. The above-mentioned storage medium is configured to store program code for performing the following steps: After receiving the raw data information to be processed from the target data source through the lake warehouse integration in the lake warehouse integration system, the method further includes: determining the data source type of the target data source through the lake warehouse integration; determining a write operator through the lake warehouse integration based on the data source type; and writing the raw data information into the raw data layer of the lake warehouse integration system through the lake warehouse integration based on the write operator. The above-mentioned storage medium is configured to store program code for performing the following steps: After processing the raw data information according to the target data pipeline through the lake warehouse integration in the lake warehouse integration system to obtain first data information, the method further includes: receiving a data query request initiated by a second target object through a third front-end interface corresponding to the data application layer in the lake warehouse integration system, and sending the data query request to the data application layer through the third front-end interface; obtaining second data information corresponding to the data query request from the lake warehouse integration layer of the lake warehouse integration system through the data application layer based on the data query request; and pushing the second data information to the third front-end interface through the data application layer.The storage medium is configured to store program code for executing the following steps: After the lake-warehouse integrated system processes the raw data information according to the target data pipeline to obtain first data information, the method further includes: constructing a training sample set according to the first data information by the data application layer; and training a neural network model for a target scenario corresponding to the first data information according to the training sample set by the data application layer to obtain a target neural model. Example 6 The present disclosure also provides a computer program product, including a computer program, which is executed by a processor to perform the above-mentioned data processing method based on the lake-warehouse integrated system. In the above-mentioned embodiments of the present disclosure, the description of each embodiment has different emphases. For portions not described in detail in a particular embodiment, reference should be made to the relevant descriptions of other embodiments. In the several embodiments provided in the present disclosure, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of the units is merely a logical functional division. In actual implementation, other division methods may be used. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not implemented. Furthermore, the coupling, direct coupling, or communication connection shown or discussed may be through interfaces, indirect coupling, or communication connection between units or modules, and may be electrical or otherwise. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the objectives of the present embodiment as needed. Furthermore, the functional units in the various embodiments of the present disclosure may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. These integrated units may be implemented in either hardware or software functional units. If these integrated units are implemented as software functional units and sold or used as independent products, they may be stored in a computer-readable storage medium.Based on this understanding, the technical solution of the present disclosure, or the portion that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product, stored in a storage medium, includes instructions for causing a computer device (such as a personal computer, server, or network device) to execute all or part of the steps of the methods described in the various embodiments of the present disclosure. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), a removable hard drive, a magnetic disk, or an optical disk. The above description is merely a preferred embodiment of the present disclosure. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present disclosure, and such improvements and modifications should also be considered within the scope of protection of the present disclosure.

Claims

Claims 1. A lake-warehouse integrated system, wherein: include: A data exploration layer, wherein the data exploration layer is used to access a target data source, construct a first custom function, and configure a target data pipeline. The target data source includes raw data information to be processed, and the target data pipeline calls the first custom function to process the raw data information; a lake-warehouse integration, wherein the lake-warehouse integration is used to execute the target data pipeline to process the raw data information to obtain first data information, and the lake-warehouse integration is also used to store the first data information; a data application layer, wherein the first object calls the first data information through the data application layer.

2. The lake-warehouse integrated system according to claim 1, wherein: The data exploration layer includes: a data source access module, wherein the data source access module accesses the target data source based on the requirements of the second object; a function construction module, wherein the function construction module constructs the first custom function based on the processing requirements of the original data information; a data pipeline configuration module, wherein the data pipeline configuration module configures the target data pipeline based on the first custom function and the processing requirements so that the target data pipeline is executed in the lake-warehouse integration.

3. The lake-warehouse integrated system according to any one of claims 1-2, wherein: The lake-warehouse integration also includes: a raw data layer, wherein the raw data layer is used to store raw data information imported from the target data source; a pipeline layer, wherein the target data pipeline is executed in the pipeline layer to process the raw data information to obtain the first data information, wherein the first data information is the data information required by the second object; a lake-warehouse integration layer, wherein the lake-warehouse integration layer is used to store the first data information.

4. The lake-storage integrated system according to any one of claims 1 to 3, wherein: The data application layer also includes: a data management module, wherein the data management module is used to manage the first data information; a data query and analysis module, wherein the data query and analysis module is used to query and analyze the first data information to obtain the data information required by the first object; and a data application module, wherein the data application module is used to process and handle the first data information to meet task requirements.

5. The lake-storage integrated system according to claim 3, wherein: The data exploration layer also includes: a metadata processing module, wherein the metadata processing module is used to detect the first data information stored in the lake-warehouse integrated layer and obtain metadata information corresponding to the first data information.

6. A data processing method based on a lake-warehouse integrated system, wherein: The data processing method is applied to the lake-warehouse integrated system described in any one of claims 1 to 5, including: receiving the original data information to be processed from the target data source through the lake-warehouse integrated system in the lake-warehouse integrated system; when there is no custom function for processing the original data information in the data exploration layer of the lake-warehouse integrated system, generating the first custom function based on the processing requirements of the original data information through the data exploration layer, and generating a target data pipeline based on the processing requirements and the first custom function, wherein the target data pipeline calls the first custom function to process the original data information; processing the original data information according to the target data pipeline through the lake-warehouse integrated system in the lake-warehouse integrated system to obtain first data information.

7. The method according to claim 6, wherein: Before receiving the raw data information to be processed from the target data source through the lake warehouse integration in the lake warehouse integration system, the method further includes: receiving the first target object through the first front-end interface corresponding to the data source access module in the data exploration layer 24 The method includes: receiving first information input by the first target object through the first front-end interface, and sending the first information to the data source access module through the first front-end interface, wherein the first information at least includes identification information of the data source; determining the target data source through the data source access module based on the target language model and the identification information in the first information, and pushing parameter information corresponding to the target data source to the first front-end interface; receiving second information input by the first target object through the first front-end interface, and establishing a connection with the target data source according to the second information through the data source access module to receive the original data information to be processed from the target data source, wherein the second information at least includes identity identification ID information of the first target object.

8. The method according to any one of claims 6 to 7, wherein: Generating the first custom function based on the processing requirements of the original data information through the data exploration layer in the lake-warehouse integrated system includes: receiving the third information input by the first target object through the second front-end interface corresponding to the function construction module in the data exploration layer, and sending the third information to the function construction module through the second front-end interface, wherein the third information is the description information of the processing requirements; judging whether it is necessary to call the target tool based on the target language model through the function construction module to obtain a judgment result; when the judgment result indicates that the target tool needs to be called, processing the third information based on the target language model and the target tool through the function construction module to obtain the first custom function; pushing the first custom function to the second front-end interface through the function construction module.

9. The method according to claim 8, wherein: After obtaining the first custom function, the method further includes: obtaining a preset custom function specification through the function construction module; verifying the first custom function according to the custom function specification through the function construction module to obtain a verification result; and if the verification result indicates that there is no abnormality in the first custom function, compiling and deploying the first custom function through the function construction module so that the target data pipeline can call the first custom function for data processing.

10. The method according to any one of claims 6 to 9, wherein: After the data exploration layer in the lake-warehouse integrated system generates the first custom function based on the processing requirements of the original data information, the method further includes: calling the node-based workflow automation tool through the function construction module in the data exploration layer to compile the first custom function to obtain the compiled first custom function; constructing a workflow based on the compiled first custom function through the function construction module, wherein the workflow is used to verify whether the first custom function meets the processing requirements; executing the workflow through the function construction module to obtain an execution result, and verifying whether the first custom function meets the processing requirements based on the execution result.

11. The method according to any one of claims 6 to 10, wherein: Generating a target data pipeline according to the processing requirements and the first custom function includes: determining, by a data pipeline configuration module in the data exploration layer, a data pipeline template based on a target language model and the processing requirements; configuring, by the data pipeline configuration module, the data pipeline according to the data pipeline template and the first custom function to obtain configuration information; and generating, by the data pipeline configuration module, the target data pipeline according to the configuration information.

12. The method according to any one of claims 6 to 11, wherein: In the lake warehouse integrated system After the body receives the original data information to be processed from the target data source, the method also includes: determining the data source type of the target data source through the lake-warehouse integration; determining a write operator through the lake-warehouse integration based on the data source type; and writing the original data information into the original data layer of the lake-warehouse integration system through the lake-warehouse integration based on the write operator.

13. The method according to any one of claims 6 to 12, wherein: After the lake-warehouse integrated system processes the original data information according to the target data pipeline to obtain the first data information, the method also includes: receiving a data query request initiated by a second target object through a third front-end interface corresponding to the data application layer in the lake-warehouse integrated system, and sending the data query request to the data application layer through the third front-end interface; obtaining the second data information corresponding to the data query request from the lake-warehouse integrated layer of the lake-warehouse integrated system through the data application layer according to the data query request; and pushing the second data information to the third front-end interface through the data application layer.

14. The method according to any one of claims 6 to 13, wherein: After the lake-warehouse integrated system processes the original data information according to the target data pipeline to obtain the first data information, the method also includes: constructing a training sample set according to the first data information through the data application layer; training the neural network model in the target scenario corresponding to the first data information according to the training sample set through the data application layer to obtain the target neural model.

15. A data processing device based on a lake-warehouse integrated system, wherein: include: A first receiving unit is configured to receive raw data information to be processed from a target data source through the lake-warehouse integration in the lake-warehouse integration system; A generation unit is used to generate the first custom function based on the processing requirements of the original data information through the data exploration layer when there is no custom function for processing the original data information in the data exploration layer of the lake-warehouse integrated system, and generate a target data pipeline based on the processing requirements and the first custom function, wherein the target data pipeline calls the first custom function to process the original data information; a processing unit is used to process the original data information based on the target data pipeline through the lake-warehouse integrated system in the lake-warehouse integrated system to obtain first data information.

16. A computer-readable storage medium, wherein: The computer-readable storage medium includes a stored program, wherein when the program is running, the device where the storage medium is located is controlled to execute the data processing method based on the lake-warehouse integrated system as described in any one of claims 6 to 14.

17. An electronic device, wherein: include: a memory storing an executable program; A processor is used to run the program, wherein when the program is running, it executes the data processing method based on the lake-warehouse integrated system as described in any one of claims 6 to 14.

18. A computer program product, wherein It includes a computer program, which, when executed by a processor, implements the data processing method based on the lake-warehouse integrated system as described in any one of claims 6 to 14.

Images