Data into lake method, device, equipment, medium and program product

Abstract

The invention provides a data lake entering method which can be applied to the technical field of big data. The data lake entering method comprises the steps that original data corresponding to a plurality of target mechanisms are acquired, and the original data carry a unique identifier of each target mechanism; according to the unique identifier of each target mechanism, the original data is divided into different database operation environments for storage, the database operation environments are pre-configured with generation rules of each target mechanism, and the generation rules are configured in the different database operation environments based on the total lake entering task; according to the generation rule, generating a lake entering task corresponding to each target mechanism, and allocating the lake entering task to a container mounted in each database operation environment for execution to obtain lake entering data corresponding to each target mechanism; and respectively storing the lake entering data corresponding to different target mechanisms to different positions in the data lake. The invention further provides a data lake entering device, equipment, a medium and a program product.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of big data, in particular to the technical field of data storage, and more particularly to a data lake entry method, device, equipment, medium and program product. BACKGROUND

[0002] At present, data lake entry, as one of important means of data processing, has gradually become an indispensable step in the development process. Data lake entry relies on data synchronization according to the entry file of the corresponding rule generated according to the original data. However, in the prior art, due to the influence of the load pressure of a single database and the system resource utilization rate, the speed of data lake entry is slow and the efficiency is low. SUMMARY

[0003] In view of the above problems, the present application provides a data lake entry method, device, equipment, medium and program product.

[0004] According to a first aspect of the present application, a data lake entry method is provided, comprising: obtaining original data corresponding to a plurality of target institutions, wherein the original data carries a unique identifier of each target institution; dividing the original data into different database running environments for storage according to the unique identifier of each target institution, wherein the database running environment is pre-configured with a generation rule of each target institution, and the generation rule is configured in different database running environments based on a total lake entry task; generating a lake entry task corresponding to each target institution according to the generation rule, and distributing the lake entry task to a container mounted under each database running environment for execution to obtain lake entry data corresponding to each target institution; and storing the lake entry data corresponding to different target institutions in different positions in the data lake.

[0005] According to an embodiment of the present application, dividing the corresponding original data into different database running environments for storage according to the unique identifier of each target institution comprises: pre-configuring a data routing mechanism, wherein the data routing mechanism is configured to automatically route the corresponding original data to different database running environments according to the unique identifier of each target institution; and routing the corresponding original data to different database running environments for storage using the data routing mechanism according to the unique identifier of each target institution.

[0006] According to an embodiment of the present application, generating a lake entry task corresponding to each target institution according to the generation rule comprises: analyzing the generation rule to obtain the business requirements of each target institution; determining the data processing logic information, task priority information and target container information of the task allocation of the original data corresponding to each target institution according to the business requirements; and generating a lake entry task corresponding to each target institution according to the data processing logic information, task priority information and target container information.

[0007] According to the embodiments of the present application, the method for assigning the lake-in tasks to the containers mounted under each database running environment for execution comprises: pre-configuring a task assignment mechanism, wherein the task assignment mechanism is configured to be capable of automatically assigning each lake-in task to a corresponding target container according to target container information; and distributing and processing the lake-in tasks in parallel by assigning the lake-in tasks to the target containers mounted under each database running environment according to the target container information.

[0008] According to the embodiments of the present application, the method for assigning the lake-in tasks to the containers mounted under each database running environment for execution further comprises: inserting the lake-in tasks into a task pool, wherein the task pool is used to cache all the lake-in tasks to be processed; obtaining load information of each target container; and distributing and processing the lake-in tasks in parallel by assigning the lake-in tasks to the target containers satisfying the load conditions according to the load information.

[0009] According to the embodiments of the present application, a plurality of containers are mounted under one database running environment; and the method for assigning the lake-in tasks to the containers mounted under each database running environment for execution further comprises: decomposing one lake-in task into a plurality of sub-tasks; assigning the plurality of sub-tasks obtained by decomposing the one lake-in task to the plurality of containers mounted under the same database running environment; and processing the plurality of sub-tasks in parallel by using the plurality of containers.

[0010] According to the embodiments of the present application, the method further comprises: classifying the lake-in data generated under each database running environment according to the unique identifier of each target institution, wherein the lake-in data belonging to the same target institution is classified into the same category; and integrating the lake-in data belonging to the same category into one lake-in file to obtain the lake-in file corresponding to each target institution.

[0011] According to the embodiments of the present application, the method further comprises: verifying the format of the lake-in data before storing the lake-in data corresponding to different target institutions to different positions in the data lake; and in response to the verification passing, transmitting the lake-in data corresponding to each target institution to the target position in the data lake for storage.

[0012] The second aspect of the present application provides a data lake entering device, comprising: an acquisition module configured to acquire original data corresponding to a plurality of target institutions, wherein the original data carries a unique identifier of each target institution; a division module configured to divide the original data into different database running environments for storage according to the unique identifier of each target institution, wherein the database running environment is pre-configured with a generation rule of each target institution, and the generation rule is configured in different database running environments based on a total lake entering task; a generation module configured to generate a lake entering task corresponding to each target institution according to the generation rule, and distribute the lake entering task to a container mounted in each database running environment for execution to obtain lake entering data corresponding to each target institution; and a storage module configured to store the lake entering data corresponding to different target institutions in different positions in the data lake.

[0013] The third aspect of the present application provides an electronic device, comprising: one or more processors; a memory configured to store one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the method.

[0014] The fourth aspect of the present application further provides a computer-readable storage medium having a computer program or instructions stored thereon, wherein the computer program or instructions are executed by a processor to implement the steps of the method.

[0015] The fifth aspect of the present application further provides a computer program product comprising a computer program or instructions, wherein the computer program or instructions are executed by a processor to implement the steps of the method. BRIEF DESCRIPTION OF DRAWINGS

[0016] The above and other objects, features and advantages of the present application will become more apparent from the following description of embodiments of the present application, taken in conjunction with the accompanying drawings, in which:

[0017] Figure 1 An application scenario diagram of a data lake entering method according to an embodiment of the present application is schematically shown;

[0018] Figure 2 A flowchart of a data lake entering method according to an embodiment of the present application is schematically shown;

[0019] Figure 3 A method flowchart for dividing original data according to an embodiment of the present application is schematically shown;

[0020] Figure 4 A method flowchart for generating a lake entering task according to an embodiment of the present application is schematically shown;

[0021] Figure 5 A method flowchart for distributing a lake entering task according to an embodiment of the present application is schematically shown;

[0022] Figure 6 A method flowchart for unified coordination and allocation of lake entry tasks by a task pool according to an embodiment of the present application is schematically shown;

[0023] Figure 7 A method flowchart for decomposed execution of lake entry tasks according to an embodiment of the present application is schematically shown;

[0024] Figure 8 A structural block diagram of a data lake entry device according to an embodiment of the present application is schematically shown; and

[0025] Figure 9 A block diagram of an electronic device suitable for implementing a data lake entry method according to an embodiment of the present application is schematically shown. DETAILED DESCRIPTION

[0026] Hereinafter, embodiments of the present application will be described with reference to the accompanying drawings. It is to be understood, however, that the description is merely exemplary of the present application, and is not intended to limit the scope of the present application. In the following detailed description of the embodiments of the present application, numerous specific details are set forth in order to provide a thorough understanding of the present application. However, it will be apparent to one skilled in the art that the present application can be practiced without these specific details. In other instances, well-known structures and functions have not been described in detail in order to avoid obscuring aspects of the present application.

[0027] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the present application. As used herein, the term "including" "comprising" and the like are meant to be inclusive, but not limiting to the components, steps, operations and / or the features that are recited.

[0028] All terms used herein, including technical and scientific terms, have the meanings commonly understood by one of ordinary skill in the art unless otherwise defined herein. It should be noted that the use of certain terms or phrases in various places in the specification is not intended to exclude those terms or phrases from the scope of the application, unless otherwise indicated.

[0029] In the case where expressions such as "at least one of A, B, and C, etc." are used, it is generally intended to mean any one of A, B, and C, etc., alone, or any combination of A, B, and C, etc., such as A or B, A or C, B or C, A and B, A and C, B and C, A, B, and C, etc.

[0030] It should be noted that the data lake entry method and device provided in the present application can be applied to the application of data lake entry in the field of financial technology, and can also be applied to the application of data lake entry in any field other than the field of financial technology. The application field of the data lake entry method and device provided in the present application is not limited.

[0031] In the technical solution of the present application, the user information (including but not limited to user personal information, user image information, user equipment information such as location information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved are information and data authorized by the user or authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of related data comply with relevant laws, regulations and standards, necessary security measures are taken, public order and good customs are not violated, and corresponding operation portals are provided for users to choose authorization or refusal.

[0032] In the scenario of using personal information for automated decision-making, the method, device and system provided in the embodiments of the present application all provide corresponding operation portals for users to choose to agree or refuse the automated decision-making result. If the user chooses to refuse, the expert decision-making process is entered. The expression "automated decision-making" here refers to the activity of automatically analyzing, evaluating the behavior habits, interests and hobbies or economic, health and credit conditions of a person through a computer program and making decisions. The expression "expert decision-making" here refers to the activity of making decisions by personnel who are engaged in a certain field, have special experience, knowledge and skills and reach a certain professional level.

[0033] The embodiments of the present application provide a data lake entry method, comprising: acquiring original data corresponding to a plurality of target institutions, wherein the original data carries a unique identifier of each target institution; dividing the original data into different database running environments for storage according to the unique identifier of each target institution, wherein the database running environment is pre-configured with a generation rule of each target institution, and the generation rule is configured based on a total lake entry task in different database running environments; generating a lake entry task corresponding to each target institution according to the generation rule, and distributing the lake entry task to a container mounted under each database running environment for execution to obtain lake entry data corresponding to each target institution; and storing the lake entry data corresponding to different target institutions in different positions in the data lake.

[0034] Figure 1 The application scenario diagram of the data lake entry method according to the embodiments of the present application is schematically shown.

[0035] As Figure 1As shown, the application scenario 100 according to this embodiment can include a first terminal device 101, a second terminal device 102, and a third terminal device 103. A network 104 is a medium for providing a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and a server 105. The network 104 can include various connection types, such as wired, wireless communication links, or optical fiber cables, and the like.

[0036] A user can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 through the network 104 to receive or send messages, and the like. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, and the like (only as examples).

[0037] The first terminal device 101, the second terminal device 102, and the third terminal device 103 can be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0038] The server 105 can be a server providing various services, such as a background management server supporting a website browsed by a user using the first terminal device 101, the second terminal device 102, and the third terminal device 103 (only as an example). The background management server can analyze and process received user requests and the like, and feed back the processing results (such as a webpage, information, or data, or the like, obtained or generated according to a user request) to the terminal device.

[0039] It should be noted that the data lake method provided by the embodiments of the present application can generally be executed by the server 105. Correspondingly, the data lake device provided by the embodiments of the present application can generally be arranged in the server 105. The data lake method provided by the embodiments of the present application can also be executed by a server or a server cluster different from the server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or the server 105. Correspondingly, the data lake device provided by the embodiments of the present application can also be arranged in a server or a server cluster different from the server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or the server 105.

[0040] It should be understood that Figure 1 The number of terminal devices, networks, and servers in the above description is only illustrative. According to the implementation needs, there can be any number of terminal devices, networks, and servers.

[0041] The following will be based on Figure 1 The described scene, through Figures 2-7 The data import method according to the embodiments of this application will be described in detail.

[0042] Figure 2 A flowchart illustrating a data import method for a lake according to an embodiment of this application is shown schematically.

[0043] like Figure 2 As shown, the data import method of this embodiment includes operations S210 to S240, and the data import method can be executed by a server.

[0044] In operation S210, raw data corresponding to multiple target institutions is obtained, wherein the raw data carries a unique identifier for each target institution.

[0045] In operation S220, the raw data is divided into different database operating environments for storage based on the unique identifier of each target institution. The database operating environment is pre-configured with the generation rules for each target institution. The generation rules are configured based on the total lake entry task in different database operating environments.

[0046] In operation S230, according to the generation rules, a lake entry task corresponding to each target institution is generated, and the lake entry task is assigned to the container mounted in each database running environment for execution, so as to obtain the lake entry data corresponding to each target institution.

[0047] In operation S240, the incoming data corresponding to different target institutions are stored in different locations in the data lake.

[0048] The goal of data ingestion into a data lake is to efficiently and reliably store the raw data of multiple target organizations (such as target organization A, target organization B, target organization C, etc.) in a designated location within the data lake, making it available for subsequent analysis and machine learning.

[0049] In the embodiments of this application, the raw data corresponding to multiple target organizations is first obtained. The raw data may include, for example, structured business data (such as transaction orders, customer relationship management records and supply chain inventory lists); semi-structured logs (such as web page access logs, application exception stacks and device event reports); and real-time streaming data (such as network traffic and mobile device tracking time).

[0050] Before obtaining raw data from multiple target organizations, the consent or authorization of these organizations can be obtained. For example, before operation S210, requests to obtain raw data can be sent to multiple target organizations. If the multiple target organizations agree or authorize the acquisition of raw data, operation S210 is executed.

[0051] The original data of each target institution is assigned a unique identification, i.e., a code, when entering the system. The code is a key to distinguish different target institutions and can be a string or a number embedded in the data stream carried by the original data.

[0052] The corresponding original data is then divided into different database running environments for storage according to the unique identification of each target institution, i.e., vertical splitting of the original data.

[0053] The database running environment, i.e., the database instance (such as instance 1, instance 2, instance 3, etc.), refers to a running database management environment.

[0054] For example, in the embodiments of the present application, the data of target institution A can be divided into instance 1, the data of target institution B can be divided into instance 2, and the data of target institution C can be divided into instance 3, etc.

[0055] During the data import process, the database running environment is used to temporarily cache the original data, providing a basis for subsequent task generation, task allocation, and task execution. Task allocation depends on the location of the original data. Only after the original data is divided into the specified database instance, the system can then generate import tasks according to the target institution identification and generation rules, and allocate the import tasks to the corresponding instance under the container for execution.

[0056] After vertical splitting of the original data, the import tasks corresponding to the original data are generated according to the pre-stored "generation rules" in the database running environment.

[0057] The generation rule is configuration information stored in the database running environment, which is generated according to the business needs of the target institution and is configured by a total import task in different database running environments. It defines how to process the original data of target institution A, target institution B, and target institution C, for example, it can include cleaning logic (such as removing invalid values, format standardization, etc.), conversion logic (such as data format conversion, field mapping, etc.), partitioning logic (such as storage partitioning according to target institution identification and time field, etc.), task allocation (such as specifying the priority of processing tasks, specifying the execution container of processing tasks, etc.), and according to the generation rule, specific import tasks can be generated.

[0058] After generating the import tasks, task allocation is performed. In the embodiments of the present application, the import tasks are allocated to the containers mounted under each database running environment, and the import tasks are executed synchronously according to the specified containers, i.e., distributed parallel processing, to generate the import data corresponding to each target institution.

[0059] Finally, the generated lake entry data is stored to the target position corresponding to the data lake to complete the data entry into the lake. The data lake usually has multiple partitions, and each partition corresponds to a target institution. In the embodiment of the present application, the respective lake entry data can be stored to the corresponding partition position according to the identifier of the target institution.

[0060] According to the embodiment of the present application, by using the unique identifier of the target institution, the original data of different target institutions is respectively stored to different database running environments (i.e. database instances), and the vertical splitting of the original data is performed, and then the generated lake entry task is distributed to the container mounted under the database running environment for distributed parallel processing. The combination of the two realizes the double-layer vertical splitting of the data layer (original data) and the computing layer (lake entry task). Since the vertical splitting of the data layer isolates the original data of multiple target institutions to be stored in independent database running environments, the load pressure of a single running environment is reduced. The computing layer distributes and concurrently executes the lake entry task in multiple containers, which significantly shortens the processing time. The coordination greatly optimizes the utilization of system resources, so that the speed and efficiency of data entry into the lake are significantly improved.

[0061] Figure 3 A method flowchart for splitting original data according to an embodiment of the present application is schematically shown.

[0062] As shown in Figure 3 the method for splitting original data of the embodiment includes operation S310~operation S320.

[0063] In operation S310, a data routing mechanism is pre-configured, wherein the data routing mechanism is configured to be able to automatically route the corresponding original data to different database running environments according to the unique identifier of different target institutions.

[0064] In operation S320, according to the unique identifier of each target institution, the data routing mechanism is used to respectively route the corresponding original data to different database running environments for storage.

[0065] In the embodiment of the present application, the original data is divided by using the pre-configured data routing mechanism.

[0066] When the original data of multiple target institutions carries the respective identifier into the system, the data routing mechanism can automatically classify the original data according to the identifier, and then route to different running environments for storage according to different categories, for example:

[0067] The data of target institution A is routed to database instance 1 and stored to the database table of instance 1.

[0068] The data of target institution B is routed to database instance 2 and stored to the database table of instance 2.

[0069] The data of the target organization C is routed to the database instance 3 and stored to the database table of the instance 3.

[0070] According to the embodiments of the present application, the data routing mechanism uses the unique identifier of the target organization as the routing basis, can accurately locate the storage location of the data, and realizes data shunting, which takes into account storage optimization and performance improvement.

[0071] Figure 4 A method flowchart for generating a lake entry task according to an embodiment of the present application is schematically shown.

[0072] As shown in Figure 4 The method for generating a lake entry task of this embodiment includes operations S410-S430.

[0073] In operation S410, the generation rule is parsed to obtain the business requirement of each target organization.

[0074] In operation S420, according to the business requirement, the data processing logic information, the task priority information and the target container information of the task allocation of the original data corresponding to each target organization are determined.

[0075] In operation S430, according to the data processing logic information, the task priority information and the target container information, the lake entry task corresponding to each target organization is generated.

[0076] In the embodiments of the present application, the lake entry task is generated according to the data processing logic, the task priority and the target container specification of the task allocation.

[0077] The lake entry task is a specific task generated by the system according to the original data of the target organization and its generation rule in the data lake entry process, which aims to process, convert or process the original data to generate output that meets the target data lake format or purpose.

[0078] After the original data is divided, the system will first read the generation rule stored in advance in the database running environment.

[0079] Then the generation rule is parsed to obtain the specific business requirement, and further according to the business requirement, the processing matters of the original data are determined, for example: data processing logic information (such as format conversion, calculation rule, etc.), task priority information (such as the execution order of the original data processing of each target organization), and target container information of task allocation (such as which task should be allocated to which target container, etc.).

[0080] Finally, the generation of the lake entry task is completed according to these processing matters.

[0081] In addition, since the lake entry task is generated according to the generation rule pre-stored in the database running environment, the rule is generated based on the business requirement of the target organization (such as data processing logic, task priority, target container for task allocation, etc.), and therefore, the system can generate one lake entry task for the raw data of one target organization, or generate multiple lake entry tasks for the raw data of one target organization. For example, if the raw data of the target organization A needs to go through multiple processing logics (such as cleaning, conversion, and aggregation), multiple lake entry tasks can be generated to process different data subsets or processing steps.

[0082] According to the embodiments of the present application, since the data processing logic, the task priority, and the target container for task allocation are all generated based on the business requirement of the target organization, the lake entry task can be generated to accurately match the requirement of each target organization, thereby reducing the risk of data deviation.

[0083] Figure 5 A method flowchart for allocating a lake entry task according to an embodiment of the present application is schematically shown.

[0084] As shown in Figure 5 , the method for allocating a lake entry task of the embodiment includes operation S510 to operation S520.

[0085] In operation S510, a task allocation mechanism is pre-configured, wherein the task allocation mechanism is configured to automatically allocate each lake entry task to the corresponding target container according to the target container information.

[0086] In operation S520, the lake entry task is allocated to the target container mounted under each database running environment for distributed parallel processing by using the task allocation mechanism according to the target container information.

[0087] In the embodiments of the present application, the pre-configured task allocation mechanism is used to guide the allocation of the lake entry task.

[0088] After the lake entry task is generated, the allocation of the lake entry task is performed next. Based on the splitting result of the raw data at the data layer, the generated lake entry task can be allocated to the container mounted under each database running environment for distributed parallel processing at the computing layer.

[0089] The allocation can be automatically completed by a task allocation mechanism, which is configured to automatically allocate each lake entry task to the corresponding target container according to the target container information in the generation rule. Since the target container information for task allocation is included in the generation rule, the task allocation mechanism can completely realize the automatic task allocation.

[0090] When each container receives the lake entry task, the lake entry task can be automatically executed.

[0091] According to the embodiment of the present application, the task allocation mechanism can automatically allocate the lake entry tasks according to the predetermined target container information, reduce manual intervention, and improve the allocation efficiency.

[0092] Figure 6 A method flowchart for uniformly coordinating and allocating lake entry tasks by a task pool according to the embodiment of the present application is schematically shown.

[0093] As Figure 6 shown, the method for uniformly coordinating and allocating lake entry tasks by the task pool according to the embodiment includes operations S610-S630.

[0094] In operation S610, the lake entry tasks are inserted into the task pool, where the task pool is used to buffer all the lake entry tasks to be processed.

[0095] In operation S620, the load information of each target container is obtained.

[0096] In operation S630, the lake entry tasks are allocated to the target containers satisfying the load condition for distributed parallel processing according to the load information.

[0097] In the embodiment of the present application, the cache function of the task pool is utilized to uniformly coordinate the allocation of the lake entry tasks in combination with the load condition of each container.

[0098] When the lake entry tasks are generated, in addition to automatically allocating the tasks by the task allocation mechanism, all the lake entry tasks can be inserted into the task pool, and then the task pool is used to uniformly allocate the tasks.

[0099] The task pool can be understood as a logical set for storing all the lake entry tasks to be processed. When all the lake entry tasks are inserted into the task pool, the system will evaluate the load information of each target container, and then the load information is combined for collaborative allocation.

[0100] For example, the lake entry tasks of the target mechanism A are allocated to container 1, but considering that the load pressure of the container 1 is relatively large at present, the lake entry tasks of the target mechanism A can be allocated to the backup container of the container 1 or other containers with smaller load pressure.

[0101] According to the embodiment of the present application, the task pool can centrally manage all the lake entry tasks to be processed, and the task pool supports the load balancing strategy, can make optimized adjustment in combination with the load condition of each container, ensures that the tasks are evenly distributed to the most suitable container, avoids single point overload, and is beneficial to improve the processing efficiency.

[0102] Figure 7 A method flowchart for decomposing and executing the lake entry tasks according to the embodiment of the present application is schematically shown.

[0103] As Figure 7 shown, the method of the embodiment for decomposing and performing the lake entry task includes operation S710 to operation S730.

[0104] In operation S710, the same lake entry task is decomposed into multiple subtasks.

[0105] In operation S720, the multiple subtasks obtained by decomposing the same lake entry task are assigned to multiple containers mounted under the same database running environment.

[0106] In operation S730, the multiple subtasks are distributed and processed in parallel by using the multiple containers.

[0107] In the embodiment of the present application, in order to further improve the efficiency of data entry into the lake, multiple containers can be mounted under a database running environment, and then the same lake entry task can be decomposed into multiple subtasks, and the multiple containers can be used to simultaneously and in parallel process multiple subtasks.

[0108] For example, a lake entry task of a target organization A is originally executed in a container mounted under a database instance 1, in order to improve efficiency, multiple containers (such as container ①, container ②, and container ③) can be mounted under the database instance 1, and then the lake entry task of the target organization A can be decomposed into multiple subtasks (such as subtask 1, subtask 2, and subtask 3), and then subtask 1 can be executed by container ①, subtask 2 can be executed by container ②, and subtask 3 can be executed by container ③, so that the three subtasks can be simultaneously and in parallel processed, and the efficiency of data entry into the lake is further improved.

[0109] According to the embodiment of the present application, on the basis of distributed and parallel processing of multiple lake entry tasks, a single lake entry task can be decomposed and then subtasks can be simultaneously and in parallel processed, so that the efficiency of data entry into the lake can be further improved.

[0110] On the basis of the above embodiment, the data entry method can further include the following operation:

[0111] According to the unique identifier of each target organization, the lake entry data generated under each database running environment is classified, wherein the lake entry data belonging to the same target organization is classified into the same category; the lake entry data belonging to the same category is integrated into one lake entry file, and the lake entry file corresponding to each target organization is obtained.

[0112] In the embodiment of the present application, the generated lake entry data is classified and arranged, and the lake entry data belonging to the same category is integrated into one lake entry file.

[0113] The original data and the generation rules stored in the same database running environment can cause scattered data (e.g., generated according to different timestamps or different processing batches), which increases the complexity of storage, retrieval and transmission. At this time, in order to ensure the logical integrity and consistency of the data, the generated lake data can be comprehensively classified and arranged, and the lake data belonging to the same category (the same target institution) is integrated into one lake file, so that the lake data of one target institution corresponds to only one file.

[0114] According to the embodiments of the present application, since the original data and the generation rules stored in the same database running environment can cause scattered data, the classification and arrangement can unify all related data of the same target institution into one file, thereby ensuring the logical integrity and consistency of the data.

[0115] On the basis of the above-mentioned embodiments, the data lake method can further include the following operations:

[0116] Before storing the lake data corresponding to different target institutions to different positions in the data lake respectively, the format of the lake data is verified; and in response to the verification passing, the lake data corresponding to each target institution is transmitted to the target position in the data lake for storage.

[0117] In the embodiments of the present application, before storing the lake data corresponding to each target institution, the judgment on the data format is strengthened.

[0118] During the generation of the lake data, the lake data can be affected by hardware factors (e.g., server overheating causing lag) or external factors (e.g., sudden interruption), etc., resulting in errors, such as format disorder, etc. Therefore, the judgment on the data format can be strengthened before the data is stored in the lake, for example, the format verification can be performed, and the data is transmitted to the target position for storage in the lake after ensuring no error.

[0119] According to the embodiments of the present application, before storing the lake data corresponding to each target institution, the data format is verified, which can detect whether the lake data has errors due to the generation or merging process, thereby avoiding storing the error data into the data lake, and helping to ensure that the data meets the expected standards of the data lake and the downstream system.

[0120] Based on the above-mentioned data lake method, the present application further provides a data lake device. The following will be described in detail Figure 8 The device is described in detail.

[0121] Figure 8 The structure block diagram of the data lake device according to the embodiments of the present application is schematically shown.

[0122] As Figure 8As shown, the data lake device 800 of the embodiment includes an acquisition module 810, a division module 820, a generation module 830, and a storage module 840.

[0123] The acquisition module 810 is configured to acquire original data corresponding to a plurality of target institutions, wherein the original data carries a unique identifier of each target institution. In an embodiment, the acquisition module 810 can be configured to perform the operation S210 described above, and details are not repeated here.

[0124] The division module 820 is configured to divide the original data into different database running environments for storage according to the unique identifier of each target institution, wherein the database running environment is pre-configured with a generation rule of each target institution, and the generation rule is configured in different database running environments based on a total lake task. In an embodiment, the determination module 820 can be configured to perform the operation S220 described above, and details are not repeated here.

[0125] The generation module 830 is configured to generate a lake task corresponding to each target institution according to the generation rule, and distribute the lake task to a container mounted in each database running environment for execution to obtain lake data corresponding to each target institution. In an embodiment, the first generation module 830 can be configured to perform the operation S230 described above, and details are not repeated here.

[0126] The storage module 840 is configured to store the lake data corresponding to different target institutions in different positions in the data lake. In an embodiment, the second generation module 840 can be configured to perform the operation S240 described above, and details are not repeated here.

[0127] According to an embodiment of the present application, the division module 820 includes a data routing mechanism configuration module and a routing module.

[0128] The data routing mechanism configuration module is configured to pre-configure a data routing mechanism, wherein the data routing mechanism is configured to automatically route the corresponding original data to different database running environments according to the unique identifier of the different target institutions.

[0129] The routing module is configured to route the corresponding original data to different database running environments for storage according to the unique identifier of each target institution by using the data routing mechanism.

[0130] According to an embodiment of the present application, the generation module 830 includes an analysis module, a determination module, and a generation sub-module.

[0131] The analysis module is configured to analyze the generation rule to obtain the business requirements of each target institution.

[0132] The determining module is configured to determine, according to a business requirement, data processing logic information, task priority information, and target container information of the target institution corresponding to the raw data.

[0133] The generating submodule is configured to generate, according to the data processing logic information, the task priority information, and the target container information, the import lake task corresponding to each target institution.

[0134] According to an embodiment of the present application, the generating module 830 further includes a task allocation mechanism configuration module and an allocation submodule.

[0135] The task allocation mechanism configuration module is configured to pre-configure a task allocation mechanism, wherein the task allocation mechanism is configured to automatically allocate each import lake task to a corresponding target container according to the target container information.

[0136] The allocation submodule is configured to allocate, according to the target container information, the import lake task to a target container mounted under each database running environment for distributed parallel processing by using the task allocation mechanism.

[0137] According to an embodiment of the present application, the generating module 830 further includes an insertion module, a load information acquisition module, and a load allocation module.

[0138] The insertion module is configured to insert the import lake task into a task pool, wherein the task pool is configured to cache all import lake tasks to be processed.

[0139] The load information acquisition module is configured to acquire load information of each target container.

[0140] The load allocation module is configured to allocate, according to the load information, the import lake task to a target container satisfying a load condition for distributed parallel processing.

[0141] According to an embodiment of the present application, the generating module 830 further includes a decomposition module, a container allocation module, and a parallel processing module.

[0142] The decomposition module is configured to decompose the same import lake task into a plurality of subtasks.

[0143] The container allocation module is configured to allocate the plurality of subtasks obtained by decomposing the same import lake task to a plurality of containers mounted under the same database running environment.

[0144] The parallel processing module is configured to perform distributed parallel processing on the plurality of subtasks by using the plurality of containers.

[0145] According to an embodiment of the present application, the data import lake device 800 further includes a classification module and an integration module.

[0146] The classification module is configured to classify the lake data generated in each database running environment according to the unique identifier of each target organization, wherein the lake data belonging to the same target organization is classified into the same category.

[0147] The integration module is configured to integrate the lake data belonging to the same category into a lake file, to obtain the lake file corresponding to each target organization.

[0148] According to the embodiments of the present application, the data lake device 800 further comprises a format verification module and a transmission module.

[0149] The format verification module is configured to verify the format of the lake data before storing the lake data corresponding to different target organizations into different positions in the data lake.

[0150] The transmission module is configured to store the lake data corresponding to each target organization into the target position in the data lake in response to the verification passing.

[0151] According to the embodiments of the present application, any one or more of the acquisition module 810, the division module 820, the generation module 830 and the storage module 840 can be combined in one module, or any one of them can be split into multiple modules. Alternatively, at least part of the function of one or more of these modules can be combined with at least part of the function of other modules, and implemented in one module. According to the embodiments of the present application, at least one of the acquisition module 810, the division module 820, the generation module 830 and the storage module 840 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on chip, a system on substrate, a system on package, an application specific integrated circuit (ASIC), or any other reasonable way of hardware or firmware that can be integrated or packaged, or any one of software, hardware and firmware or any appropriate combination of several of them. Alternatively, at least one of the acquisition module 810, the division module 820, the generation module 830 and the storage module 840 can be at least partially implemented as a computer program module which can perform the corresponding function when running.

[0152] Figure 9 The block diagram of the electronic device suitable for implementing the data lake method according to the embodiments of the present application is schematically shown.

[0153] As Figure 9As shown, the electronic device 900 according to an embodiment of the present application includes a processor 901 which can perform various appropriate actions and processes according to a program stored in a read only memory (ROM) 902 or a program loaded from a storage section 908 into a random access memory (RAM) 903. The processor 901 can include, for example, a general purpose microprocessor (e.g., a CPU), an instruction set processor, and / or a related chipset, and / or a special purpose microprocessor (e.g., an application specific integrated circuit (ASIC)), and so on. The processor 901 can also include an on-board memory for cache use. The processor 901 can include a single processing unit or multiple processing units for executing different actions of the method processes according to embodiments of the present application.

[0154] In the RAM 903, various programs and data required for the operation of the electronic device 900 are stored. The processor 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. The processor 901 performs various operations of the method processes according to embodiments of the present application by executing the programs in the ROM 902 and / or the RAM 903. Note that the programs can also be stored in one or more memories other than the ROM 902 and the RAM 903. The processor 901 can also perform various operations of the method processes according to embodiments of the present application by executing the programs stored in the one or more memories.

[0155] According to embodiments of the present application, the electronic device 900 can also include an input / output (I / O) interface 905 which is also connected to the bus 904. The electronic device 900 can also include one or more of the following components connected to the input / output (I / O) interface 905: an input section 906 including a keyboard, a mouse, etc.; an output section 907 including a display such as a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 908 including a hard disk, etc.; and a communication section 909 including a network interface card such as a LAN card, a modem, etc. The communication section 909 performs communication processing via a network such as the Internet. A drive 910 is also connected to the input / output (I / O) interface 905 as necessary. A removable medium 911 such as a magnetic disk, a magneto-optical disk, a semiconductor memory, etc. is attached to the drive 910 as necessary, so that a computer program read out from the removable medium 911 is installed in the storage section 908 as necessary.

[0156] The application further provides a computer readable storage medium, which can be included in the device / apparatus / system described in the above embodiments, or can exist independently without being assembled into the device / apparatus / system. The computer readable storage medium carries one or more programs, which, when executed, implement the method according to the embodiments of the application.

[0157] According to the embodiments of the application, the computer readable storage medium can be a non-volatile computer readable storage medium, which can include, but is not limited to, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any appropriate combination thereof. In this application, a computer readable storage medium can be any tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. For example, according to the embodiments of the application, the computer readable storage medium can include one or more of the above-described ROM 902 and / or RAM 903 and / or one or more memories other than the ROM 902 and the RAM 903.

[0158] The embodiments of the application also include a computer program product, which includes a computer program containing program codes for executing the methods shown in the flowcharts. When the computer program product is run in a computer system, the program codes are used to make the computer system implement the methods provided by the embodiments of the application.

[0159] The above-described functions defined in the system / apparatus of the embodiments of the application are performed when the computer program is executed by the processor 901. According to the embodiments of the application, the above-described system, apparatus, module, unit, etc. can be implemented by computer program modules.

[0160] In one embodiment, the computer program can rely on a tangible storage medium such as an optical storage medium, a magnetic storage medium, etc. In another embodiment, the computer program can also be transmitted, distributed, and downloaded in the form of a signal on a network medium, and be downloaded and installed through the communication part 909, and / or installed from the detachable medium 911. The program codes contained in the computer program can be transmitted by any appropriate network medium, including but not limited to wireless, wired, etc., or any appropriate combination thereof.

[0161] In such embodiments, the computer program can be downloaded and installed from the network via the communication section 909, and / or installed from the removable media 911. When the computer program is executed by the processor 901, the above-described functions defined in the system of the embodiments of the present application are performed. According to the embodiments of the present application, the system, device, apparatus, module, unit, and the like described above can be implemented by the computer program modules.

[0162] According to the embodiments of the present application, the program code for executing the computer program provided by the embodiments of the present application can be written in any combination of one or more programming languages, and specifically, these computer programs can be implemented using high-level procedural and / or object-oriented programming language, and / or assembly / machine language. The programming language includes, but is not limited to, such as Java, C++, python, "C" language, or similar programming language. The program code can be executed entirely on the user computing device, partially on the user device, partially on a remote computing device, or entirely on a remote computing device or server. In the case involving a remote computing device, the remote computing device can be connected to the user computing device through any kind of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, connected to the Internet through an Internet service provider).

[0163] The flowcharts and block diagrams in the drawings illustrate the possible architectures, functions, and operations of systems, methods, and computer program products according to various embodiments of the present application. In this regard, each block in the flowcharts or block diagrams can represent a module, a program segment, or a portion of code, which contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur in a different order than that shown in the figures. For example, two blocks noted in succession can actually be executed substantially concurrently, or they can sometimes be executed in reverse order, depending on the functionality involved. It should also be noted that each block in the flowcharts or block diagrams, and combinations of blocks in the flowcharts or block diagrams, can be implemented by dedicated hardware-based systems that perform the specified functions or operations, or can be implemented by a combination of dedicated hardware and computer instructions.

[0164] Those skilled in the art can understand that the features described in various embodiments of the present application can be combined and / or integrated in various combinations and / or integrations, even if such combinations or integrations are not explicitly described in the present application. In particular, the features described in various embodiments of the present application can be combined and / or integrated in various combinations and / or integrations without departing from the spirit and teachings of the present application. All such combinations and / or integrations fall within the scope of the present application.

Claims

1. A method of data lake-ing, characterized by, The method comprises: obtaining original data corresponding to a plurality of target institutions, wherein the original data carries a unique identifier of each target institution; storing the original data in different database running environments according to the unique identifier of each target institution, wherein the database running environments are pre-configured with generation rules of each target institution, and the generation rules are configured in different database running environments based on total lake entry tasks; generating lake entry tasks corresponding to each target institution according to the generation rules, and distributing the lake entry tasks to containers mounted under each database running environment for execution to obtain lake entry data corresponding to each target institution; storing the lake entry data corresponding to different target institutions in different locations in the data lake.

2. The method of claim 1, wherein, The original data corresponding to each target institution is respectively divided and stored in different database running environments according to the unique identifier of each target institution, comprising: pre-configuring a data routing mechanism, wherein the data routing mechanism is configured to automatically route the original data corresponding to different target institutions to different database running environments according to the unique identifier of each target institution; routing the original data corresponding to each target institution to different database running environments for storage according to the unique identifier of each target institution by using the data routing mechanism.

3. The method of claim 1, wherein, The generation rules of the lake entry tasks corresponding to each target institution comprise: parsing the generation rules to obtain business requirements of each target institution; determining data processing logic information, task priority information and target container information for task allocation of the original data corresponding to each target institution according to the business requirements; generating lake entry tasks corresponding to each target institution according to the data processing logic information, the task priority information and the target container information.

4. The method of claim 3, wherein, The lake entry tasks are distributed to containers mounted under each database running environment for execution, comprising: pre-configuring a task allocation mechanism, wherein the task allocation mechanism is configured to automatically allocate each lake entry task to the corresponding target container according to the target container information; distributing the lake entry tasks to target containers mounted under each database running environment for distributed parallel processing according to the target container information by using the task allocation mechanism.

5. The method of claim 4, wherein, The lake entry tasks are distributed to containers mounted under each database running environment for execution, further comprising: inserting the lake entry tasks into a task pool, wherein the task pool is used to cache all lake entry tasks to be processed; obtaining load information of each target container; distributing the lake entry tasks to target containers meeting the load conditions for distributed parallel processing according to the load information.

6. The method of claim 4, wherein, A plurality of containers are mounted under one database running environment; The lake entry tasks are distributed to containers mounted under each database running environment for execution, further comprising: decomposing the same lake entry task into a plurality of subtasks; distributing the plurality of subtasks obtained by decomposing the same lake entry task to a plurality of containers mounted under the same database running environment; performing distributed parallel processing on the plurality of subtasks by using the plurality of containers.

7. The method of claim 4, wherein, The method further comprises: According to the unique identifier of each target organization, the generated lake entry data under each database running environment is classified, wherein the lake entry data belonging to the same target organization is classified into the same category; The lake entry data belonging to the same category is integrated into a lake entry file, and a lake entry file corresponding to each target organization is obtained.

8. The method of claim 1, wherein, The method further comprises: Before storing the lake entry data corresponding to different target organizations to different positions in the data lake respectively, the format of the lake entry data is verified; In response to the verification passing, the lake entry data corresponding to each target organization is transmitted to the target position in the data lake for storage.

9. A data asset tag management apparatus, characterized by, The device comprises: An acquisition module is configured to acquire original data corresponding to a plurality of target organizations, wherein the original data carries a unique identifier of each target organization; A division module is configured to divide the original data into different database running environments according to the unique identifier of each target organization, wherein the database running environment is pre-configured with a generation rule of each target organization, and the generation rule is configured based on a total lake entry task in different database running environments; A generation module is configured to generate a lake entry task corresponding to each target organization according to the generation rule, and distribute the lake entry task to a container mounted under each database running environment for execution, to obtain lake entry data corresponding to each target organization; A storage module is configured to store the lake entry data corresponding to different target organizations to different positions in the data lake respectively.

10. An electronic device comprising: one or more processors; a memory for storing one or more computer programs, characterized in that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1-8.

11. A computer readable storage medium having stored thereon a computer program or instructions, characterized in that, The computer program or instructions are executed by the processor to implement the steps of the method according to any one of claims 1-8.

12. A computer program product comprising computer programs or instructions, characterized in that, The computer program or instructions are executed by the processor to implement the steps of the method according to any one of claims 1-8. The computer program or instructions are executed by the processor to implement the steps of the method according to any one of claims 1-8.

Images