Data lake updating method, device, equipment and medium based on artificial intelligence

Abstract

Embodiments of the present invention relate to the financial field and the smart medical technology field, and disclose an artificial intelligence-based data lake update method, device, equipment and medium, the method comprising: obtaining target data, the target data comprising update data; identifying the number of data rows of target update data for a target primary key in the update data, if the number of data rows is greater than or equal to a preset number of rows, merging the target update data to obtain merged data, and then obtaining a merged event time of a merged event corresponding to the merged data; extracting historical data of the target primary key in the data lake to be updated, and obtaining the historical event time of the historical event corresponding to the historical data; comparing the merged event time with the historical event time to obtain a comparison result, and determining benchmark data and data to be supplemented according to the comparison result; completing the update of the data lake to be updated according to the benchmark data and the supplemented data.

Description

Technical Field

[0001] The present invention relates to the fields of artificial intelligence, finance and smart medical technology, and in particular to an artificial intelligence-based data lake updating method, device, equipment and medium. Background Art

[0002] Data lake is a new technology for big data applications. It organizes data in a new way to implement row-level updates and deletions that traditional data warehouses do not have. Among them, the data lake architecture (Apache Hudi) is a relatively feature-rich open source data lake architecture. Usually, during the Apache Hudi data entry and update process, the file where the data is located is first located, and then the hit file is read into the memory, the corresponding data is replaced in the entire line, and then the entire file is rewritten to the distributed file system (HDFS). The default logic can implement the full-field CDC log data update entry function.

[0003] Data allows all original data (structured, semi-structured, unstructured, etc.) to be stored in the data lake. The amount of data stored in the data lake is huge and the sources are diverse. The data lake supports heterogeneous and diverse storage. In the financial and smart medical fields, as the application of semi-structured and unstructured data is increasingly valued, data lakes that allow all original data (structured, semi-structured, unstructured, etc.) to be stored gradually replace data warehouses that need to define data storage models and are only for structured data.

[0004] In the context of big data, with the intelligent development and digital application of medical health, the efficient integration of multi-source heterogeneous data faces more diverse and flexible demands and severe challenges. In the traditional medical data integration solution, it is necessary to consider the diversity of medical data and store medical data of different formats in data warehouses of different themes. Secondly, when the data table format of medical data changes, the structure of the stored data is difficult to adjust quickly, and the data table of the new framework needs to be rebuilt, resulting in increased medical data integration costs. At the same time, the efficiency of medical data integration is low and cannot meet actual needs. Therefore, data lakes are increasingly widely used in the field of smart medical care because they allow the full storage of original data (structured, semi-structured, unstructured, etc.).

[0005] In the financial field, with the acceleration of the digital transformation of the domestic banking industry, the digital transformation of the entire bank has entered the deep water zone, and the proportion of online, mobile and scenario-based business has become increasingly higher. Correspondingly, it has also brought about problems such as explosive growth in data scale and diversity of data types. Artificial intelligence technologies based on human-computer interaction, knowledge graphs, etc. have put forward new requirements for the storage, feature extraction and data processing of semi-structured and unstructured data. A single data warehouse technology can no longer cope with the above challenges. Therefore, data lakes are increasingly widely used in the financial field because they allow the full storage of original data (structured, semi-structured, unstructured, etc.).

[0006] To support real-time computing of big data, it is a common scenario to synchronize the structured data replication software (OGG) log of the log from the database system (Oracle) to the open source stream processing platform (Kafka). In order to optimize the consumption of resources such as network bandwidth and data storage, the database administrator (DBA) usually adjusts the Oracle strategy for generating OGG logs. The OGG log only contains the fields that have changed in each row of data, and does not contain the fields that have not been updated. In addition to the OGG scenario, there are other similar scenarios where the fields of the change data capture (CDC) log are incomplete, which are uniformly classified as non-full-field logs. Since Apache Hudi uses the whole row overwrite update in the data update phase, directly writing the non-full-field OGG data into Apache Hudi in the above data scenarios will result in field data loss, which will cause the problem that the data lake update cannot be carried out. Summary of the invention

[0007] In view of this, the present invention provides an artificial intelligence-based data lake update method, device, equipment and medium, which are used to solve the problem in the prior art that when writing non-full-field OGG data into Apache Hudi, field data will be lost, which makes the data lake update impossible.

[0008] To achieve one or part or all of the above purposes or other purposes, the present invention proposes an artificial intelligence-based data lake update method, including: obtaining target data, the target data including update data and primary key data targeted by the update data, the target data being used for updating the data lake;

[0009] Identify the number of data rows of target update data for a target primary key in the update data, and if the number of data rows is greater than or equal to a preset number of rows, merge the target update data to obtain merged data, and then obtain a merge event time of a merge event corresponding to the merged data;

[0010] Extracting historical data of the target primary key in the data lake to be updated, and obtaining historical event time of historical events corresponding to the historical data;

[0011] Comparing the combined event time with the historical event time to obtain a comparison result, and determining the baseline data and the supplementary data according to the comparison result;

[0012] The update of the data lake to be updated is completed according to the benchmark data and the supplementary data.

[0013] On the other hand, the present application provides a data lake updating device based on artificial intelligence, the device comprising:

[0014] A data acquisition module is used to obtain target data, where the target data includes update data and primary key data targeted by the update data, and the target data is used to update the data lake;

[0015] a merging module, used to identify the number of data rows of target update data for a target primary key in the update data, and if the number of data rows is greater than or equal to a preset number of rows, merge the target update data to obtain merged data, and then obtain a merge event time of a merge event corresponding to the merged data;

[0016] An extraction module, used to extract historical data of the target primary key in the data lake to be updated, and obtain the historical event time of the historical event corresponding to the historical data;

[0017] A comparison module, used for comparing the combined event time with the historical event time to obtain a comparison result, and determining the benchmark data and the supplementary data according to the comparison result;

[0018] An updating module is used to complete the update of the to-be-updated data lake according to the benchmark data and the supplementary data.

[0019] On the other hand, the present application provides an electronic device, including: a processor, a memory, and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the memory through the bus, and when the machine-readable instructions are executed by the processor, target data is acquired, wherein the target data includes update data and primary key data targeted by the update data, and the target data is used for updating a data lake;

[0020] Identify the number of data rows of target update data for a target primary key in the update data, and if the number of data rows is greater than or equal to a preset number of rows, merge the target update data to obtain merged data, and then obtain a merge event time of a merge event corresponding to the merged data;

[0021] Extracting historical data of the target primary key in the data lake to be updated, and obtaining historical event time of historical events corresponding to the historical data;

[0022] Comparing the combined event time with the historical event time to obtain a comparison result, and determining the baseline data and the supplementary data according to the comparison result;

[0023] The update of the data lake to be updated is completed according to the benchmark data and the supplementary data.

[0024] On the other hand, the present application provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, target data is acquired, wherein the target data includes update data and primary key data for which the update data is targeted, and the target data is used for updating a data lake;

[0025] Identify the number of data rows of target update data for a target primary key in the update data, and if the number of data rows is greater than or equal to a preset number of rows, merge the target update data to obtain merged data, and then obtain a merge event time of a merge event corresponding to the merged data;

[0026] Extracting historical data of the target primary key in the data lake to be updated, and obtaining historical event time of historical events corresponding to the historical data;

[0027] Comparing the combined event time with the historical event time to obtain a comparison result, and determining the baseline data and the supplementary data according to the comparison result;

[0028] The update of the data lake to be updated is completed according to the benchmark data and the supplementary data.

[0029] Implementing the embodiments of the present invention will have the following beneficial effects:

[0030] By acquiring the target data, the target data includes the update data and the primary key data targeted by the update data; identifying the number of data rows of the target update data for the target primary key in the update data, if the number of data rows is greater than or equal to the preset number of rows, the target update data is merged to obtain the merged data, and then the merged event time of the merged event corresponding to the merged data is obtained; extracting the historical data of the target primary key in the data lake to be updated, and obtaining the historical event time of the historical event corresponding to the historical data; comparing the merged event time with the historical event time to obtain the comparison result, and determining the benchmark data and supplementary data according to the comparison result; completing the update of the data lake to be updated according to the benchmark data and the supplementary data, it can support non-full-field data not limited to OGG type entering the lake, which greatly expands the scope of Hudi use scenarios. And sorting according to event time, without relying on the measurement mechanism (Flinkwatermark) to sacrifice timeliness, solves the problem of data reverse order. After implementing non-full-field entry into the lake, the relational database extracts CDC logs no longer need to extract data in a full-field manner, effectively reducing the network bandwidth of data from the relational database to Kafka. It also reduces the storage capacity of CDC logs in Kafka, and reduces Kafka disk and memory consumption. That is, when programs and applications related to the smart medical or financial fields are updated, there is no need to call the complete installation package of the program or application for transmission, which reduces the computer resources occupied by the programs and applications related to the smart medical or financial fields when they are updated, and also reduces the update time of the programs and applications related to the smart medical or financial fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0032] in:

[0033] Figure 1 This is an application scenario diagram of an artificial intelligence-based data lake update method provided in an embodiment of the present application;

[0034] Figure 2 This is a flow chart of an artificial intelligence-based data lake update method provided in an embodiment of the present application;

[0035] Figure 3 It is a structural diagram of an artificial intelligence-based data lake updating device provided in an embodiment of the present application;

[0036] Figure 4 is a schematic diagram of a structure of a computer device in one embodiment of the present invention;

[0037] Figure 5 It is another structural schematic diagram of a computer device in one embodiment of the present invention.

[0038] Figure 6 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application;

[0039] Figure 7 It is a structural schematic diagram of a storage medium provided in an embodiment of the present application. DETAILED DESCRIPTION

[0040] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0041] See also Figure 1 , Figure 1 1 is a block diagram of a computer device according to an exemplary embodiment. The computer device is deployed with a von Neumann architecture, for example, the computer device may be a desktop computer, a notebook computer, a server, etc.

[0042] It should be noted that this type of computer device is only an example adapted to the present invention and cannot be considered to provide any limitation on the scope of use of the present invention. This type of computer device cannot be interpreted as requiring dependence on or having Figure 1 One or more components of the exemplary computer device 200 shown in FIG.

[0043] The hardware structure of the computer device 200 may vary greatly due to differences in configuration or performance, such as Figure 1 As shown, the computer device 200 includes: a power supply 210 , an interface 230 , at least one memory 250 , and at least one central processing unit (CPU) 270 .

[0044] Specifically, the power supply 210 is used to provide operating voltage for each hardware device on the computer device 200 .

[0045] The interface 230 includes at least one wired or wireless network interface 231 , at least one serial-to-parallel conversion interface 233 , at least one input-output interface 235 , and at least one USB interface 237 , etc., for communicating with external devices.

[0046] The memory 250 is a carrier for storing resources, which may be a read-only memory, a random access memory, a disk or an optical disk, etc. The resources stored thereon include an operating system 251, an application 253 and data 255, etc. The storage method may be temporary storage or permanent storage.

[0047] The operating system 251 is used to manage and control the hardware devices and application programs 253 on the computer device 200 to enable the central processing unit 270 to calculate and process the massive data 255. The operating system 251 can be any existing operating system.

[0048] The application program 253 is a computer program that performs at least one specific task based on the operating system 251, and may include at least one module ( Figure 1 (not shown in the figure), each module may respectively contain a series of computer-readable instructions for the computer device 200.

[0049] Data 255 may be photos, pictures, etc. stored in a disk.

[0050] The central processing unit 270 may include one or more processors and is configured to communicate with the memory 250 via a bus for computing and processing the mass data 255 in the memory 250 .

[0051] As described in detail above, the computer device 200 applicable to the present invention will complete the acquisition of target data in the form of a series of computer-readable instructions stored in the memory 250 through the central processor 270, wherein the target data includes update data and primary key data targeted by the update data; identify the number of data rows of target update data for the target primary key in the update data, and if the number of data rows is greater than or equal to the preset number of rows, merge the target update data to obtain merged data, and then obtain the merged event time of the merged event corresponding to the merged data; extract the historical data of the target primary key in the data lake to be updated, and obtain the historical event time of the historical event corresponding to the historical data; compare the merged event time with the historical event time to obtain a comparison result, and determine the baseline data and supplementary data according to the comparison result; and complete the update of the data lake to be updated according to the baseline data and the supplementary data.

[0052] For example, a data lake is a centralized repository that allows for the storage of all structured and unstructured data at any scale. Data can be stored as is, without first structuring it, and run different types of analysis - from dashboards and visualizations to big data processing, real-time analysis, and machine learning to guide better decision making. Furthermore, when data lakes are applied in the financial field, they facilitate big data processing, real-time analysis, and machine learning in the financial field to guide better decision making. Similarly, when data lakes are applied in the health care field, they facilitate the development of the health care field.

[0053] like Figure 2 As shown, the embodiment of the present application provides a data lake update method based on artificial intelligence, including:

[0054] S101. Obtain target data, where the target data includes update data and primary key data targeted by the update data, and the target data is used to update the data lake;

[0055] The data lake update method provided in the embodiment of the present application can be applied to a data lake update device or a data lake update engine in various scenarios. The data lake update process is usually implemented by a server, and the server that performs the data lake update can transmit data with the user's client in real time. For example, when the server receives the data lake update request information from the client, it obtains the target data, and the target data includes the updated data and the primary key data for which the updated data is targeted.

[0056] Exemplarily, target data is obtained, and the target data includes update data and primary key data targeted by the update data. For example, if the update data is to modify operating parameters, then the modified operating parameters are the update data, and the primary key that affects the operating parameters is the primary key targeted by the update data, and the identity identification information of the primary key is used as the primary key data.

[0057] Exemplarily, when the target data is used to update programs and applications related to the smart medical field, the update data is to modify the operating parameters of the programs and applications related to the smart medical field. The modified operating parameters are the updated data, and the primary key that affects the operating parameters is the primary key targeted by the update data, and the identity identification information of the primary key is used as the primary key data.

[0058] S102, identifying the number of data rows of target update data for a target primary key in the update data, and if the number of data rows is greater than or equal to a preset number of rows, merging the target update data to obtain merged data, and then obtaining a merge event time of a merge event corresponding to the merged data;

[0059] For example, since multiple updates are often performed simultaneously during the update process, for example, updating operating parameters and updating calculation methods are performed simultaneously, when updating operating parameters, the update can be completed through a single row of data, but when updating the calculation method, a single row of data cannot meet the update requirements.

[0060] Exemplarily, the number of data rows of the target update data for the target primary key in the update data is identified. If the number of data rows is greater than or equal to the preset number of rows, the data is matched column by column, and the default field is filled with old data to achieve memory merging of non-full-field logs.

[0061] S103, extracting historical data of the target primary key in the data lake to be updated, and obtaining historical event time of historical events corresponding to the historical data;

[0062] Exemplarily, historical data of the target primary key is extracted from the data lake to be updated, that is, old data before update, and the event time is the time when each independent event occurs on its generating device.

[0063] S104, comparing the combined event time with the historical event time to obtain a comparison result, and determining benchmark data and supplementary data according to the comparison result;

[0064] For example, in the data update phase, the data is matched column by column to see if it is missing, and the old data is used for filling, so as to implement non-full-field update of Hudi data.

[0065] S105, completing the updating of the to-be-updated data lake according to the benchmark data and the supplementary data;

[0066] By acquiring the target data, the target data includes the update data and the primary key data targeted by the update data; identifying the number of data rows of the target update data for the target primary key in the update data, if the number of data rows is greater than or equal to the preset number of rows, the target update data is merged to obtain the merged data, and then the merged event time of the merged event corresponding to the merged data is obtained; extracting the historical data of the target primary key in the data lake to be updated, and obtaining the historical event time of the historical event corresponding to the historical data; comparing the merged event time with the historical event time to obtain the comparison result, and determining the benchmark data and supplementary data according to the comparison result; completing the update of the data lake to be updated according to the benchmark data and the supplementary data, it can support non-full-field data not limited to OGG type entering the lake, which greatly expands the scope of Hudi use scenarios. And sorting according to event time, without relying on the measurement mechanism (Flinkwatermark) to sacrifice timeliness, solves the problem of data reverse order. After implementing non-full-field entry into the lake, the relational database extracts CDC logs no longer need to extract data in a full-field manner, effectively reducing the network bandwidth of data from the relational database to Kafka. It also reduces the storage capacity of CDC logs in Kafka, and reduces Kafka disk and memory consumption. That is, when programs and applications related to the smart medical or financial fields are updated, there is no need to call the complete installation package of the program or application for transmission, which reduces the computer resources occupied by the programs and applications related to the smart medical or financial fields when they are updated, and also reduces the update time of the programs and applications related to the smart medical or financial fields.

[0067] In a possible implementation, the step of identifying the number of data rows of target update data for a target primary key in the update data, and if the number of data rows is greater than or equal to a preset number of rows, merging the target update data to obtain merged data, and then obtaining a merged event time of a merged event corresponding to the merged data, includes:

[0068] Identify the number of data rows of target update data for a target primary key in the update data, and if the number of data rows is greater than or equal to a preset number of rows, obtain a target update event corresponding to each target update data, and then obtain a target update event time of the target update event according to the target update event;

[0069] Arrange each target update data according to the target update event time to obtain a data queue;

[0070] The merged data is obtained based on the data queue, and then the merged event time of the merged event corresponding to the merged data is obtained.

[0071] For example, if the same primary key corresponds to multiple updated data, the preset number of rows is set to 2. If the number of updated data rows corresponding to the same primary key is greater than 2, the multiple updated data corresponding to the same primary key are sorted according to event time to obtain a data queue. Multiple updated data corresponding to the same primary key are merged by column-by-column matching, which can support non-full-field data not limited to OGG type to enter the data lake.

[0072] For example, when applications in the smart medical or financial fields need to be updated, in order to reduce the resource usage during data transmission, the updated data often only includes the data that has changed. For example, the display page of the application was white before the update and the display page was green after the update. The updated data only includes the data that changed the color of the page. The updated data is specifically the underlying code. When there are multiple rows of data corresponding to the same primary key, the multiple updated data corresponding to the same primary key are merged through column-by-column matching, that is, the complete updated data of the primary key is obtained.

[0073] In a possible implementation, the step of obtaining the merged data based on the data queue, and then acquiring the merged event time of the merged event corresponding to the merged data, includes:

[0074] Using the second target update data in the data queue as the reference data, using the first target update data in the data queue as the supplementary data, and using the remaining target update data in the data queue as the candidate target update data;

[0075] Determine whether the benchmark data has any default by traversing the benchmark data, and if the benchmark data has any default, obtain the positioning data of the default position on the benchmark data;

[0076] Extracting target data from the supplementary data based on the positioning data, and merging the target data with the reference data to obtain first merged data;

[0077] Merged data is obtained according to the first merged data and the candidate target update data, and then a merge event time of a merge event corresponding to the merged data is obtained.

[0078] Exemplarily, the second data in the data queue is extracted as the reference data, and the first data is used as the supplementary data. Each field of the reference data is traversed one by one, and it is determined whether the field is missing. If the reference data field is missing, the corresponding field is extracted from the supplementary data and filled into the reference data. Since the event time is proportional to the amount of data, the second data is used as the reference data and the first data is used as the supplementary data to ensure that the data with the largest amount of data is always used as the supplementary data to avoid data omission.

[0079] For example, when applications in the smart medical or financial fields need to be updated, the specific update content of different rows of data is different, and thus the amount of data (the number of characters in the code) required to implement the update process is also different. Therefore, the data with the largest amount of data is always used as supplementary data to avoid data omissions.

[0080] In a possible implementation manner, the step of obtaining merged data according to the first merged data and the candidate target update data, and then acquiring a merged event time of a merged event corresponding to the merged data, includes:

[0081] Using the first combined data as supplementary data and the candidate target updated data as reference data;

[0082] Determine whether the benchmark data has any default by traversing the benchmark data, and if the benchmark data has any default, obtain the positioning data of the default position on the benchmark data;

[0083] Target data is extracted from the supplementary data based on the positioning data, and the target data is merged with the reference data to obtain merged data, and then a merged event time of a merged event corresponding to the merged data is obtained.

[0084] Exemplarily, after obtaining the first merged data, the first merged data is used as supplementary data, and then the third data (alternative data) is extracted as the benchmark data, and the above judgment and filling process is repeated. According to the above method, all data with the same primary key are traversed and supplemented in sequence. The updated data for the same primary key are merged by matching column by column, and the default field is filled with the old data, that is, the data that remains unchanged before and after the update, to realize the memory merging of non-full-field logs.

[0085] For example, when applications in the field of smart healthcare or finance need to be updated, a primary key often corresponds to multiple functions (contents). When one of the functions is updated, if the other functions of the primary key do not change, the updated data does not involve the underlying code of other functions. When stored in the data lake, the complete data packet of the application in the field of smart healthcare or finance needs to be stored. Therefore, it is necessary to use old data, that is, data that remains unchanged before and after the update, to fill in the updated data and realize the memory merging of non-full-field logs, so that the data lake can complete the storage of applications in the field of smart healthcare or finance.

[0086] In a possible implementation manner, after the step of identifying the number of data rows of target update data for a target primary key in the update data, the method further includes:

[0087] If the number of data rows is less than the preset number of rows, the target updated data is used as the merged data, and the step of obtaining the merged event time of the merged event corresponding to the merged data is continued.

[0088] Exemplarily, if the number of data rows is less than the preset number of rows, for example, in a parameter update process that can be completed by updating data in a single row, the updated data in the single row is used as the merged data, and the merged event time of the merged event corresponding to the merged data is continuously obtained. This avoids the risk of data duplication caused by merging the updated data in a single row, thereby causing the update to fail.

[0089] For example, when an application in the field of smart healthcare or finance needs to be updated, when the number of data rows of the target update data for the target primary key is less than the preset number of rows, the updated data of a single row is used as merged data to avoid merging the updated data of a single row to cause data duplication, which in turn leads to confusion in the underlying code and the risk of the application in the field of smart healthcare or finance not being able to operate normally.

[0090] In a possible implementation manner, the step of comparing the combined event time with the historical event time to obtain a comparison result, and determining the benchmark data and the supplementary data according to the comparison result includes:

[0091] Comparing the merged event time with the historical event time to obtain a comparison result, the comparison result comprising a first result indicating that the merged event time is greater than the historical event time, a second result indicating that the merged event time is less than the historical event time, and a third result indicating that the merged event time is equal to the historical event time;

[0092] When the comparison result is the first result, the historical data is used as supplementary data and the combined data is used as reference data;

[0093] When the comparison result is the second result, the historical data is used as the reference data, and the combined data is used as the supplementary data;

[0094] When the comparison result is the third result, the historical data is used as supplementary data, and the combined data is used as reference data.

[0095] For example, in theory, the event time of the newly written data should be greater than the event time of the same primary key in the existing data. To be compatible with data inversion, it is necessary to determine whether the event time of the data after the memory merge is greater than the event time in the existing data. The data with a smaller event time is used as the data to be supplemented, and the data with a larger event time is used as the baseline data. If the two data times are the same, the data after the memory merge is used as the baseline data, and the existing data is used as the data to be supplemented.

[0096] For example, by being compatible with data inversion, it is ensured that the update process of applications in the smart medical or financial fields proceeds normally, and it is also ensured that applications in the smart medical or financial fields can be correctly updated.

[0097] In a possible implementation, the step of identifying the number of data rows of target update data for a target primary key in the update data includes:

[0098] Randomly select any primary key data from the primary key data targeted by the update data as the target primary key data, and use the primary key corresponding to the target primary key data as the target primary key;

[0099] The target primary key data is matched with the primary key data targeted by the update data, and the number of data rows of the target update data targeted at the target primary key in the update data is identified according to the matching result.

[0100] Exemplarily, since multiple updates are often performed simultaneously during the update process, for example, the update of operating parameters and the update of calculation methods are performed simultaneously, and the primary keys targeted by the update operating parameters and the update calculation method are obviously different, the primary key identification information (primary key data) of the primary key is used to confirm the primary key targeted by the single-row update data and the primary key targeted by the multiple-row update data. Differentiating the multiple updates is conducive to setting and detecting the update process, that is, the update of operating parameters and the update of calculation methods are two types of updates, and the update of operating parameters and the update of calculation methods can be processed in parallel. When the update calculation method with multiple rows of update data merges the multiple rows of update data, the update of the update operating parameters is performed.

[0101] Exemplarily, any primary key data A is randomly selected from the primary key data targeted by the update data as the target primary key data, and the primary key data targeted by the update data include primary key data A, primary key data A, primary key data B and primary key data C. The target primary key data is matched with the primary key data targeted by the update data, and it is confirmed that at least two update data are targeted at the primary key data A, then it is confirmed that the number of data rows of the target update data for the target primary key is 2.

[0102] For example, when an application in the smart medical or financial field needs to be updated, the primary key of the display function is randomly selected from the primary key data targeted by the update data as the target primary key data, the primary key of the display function is matched with the primary key data targeted by the update data, and at least two update data are confirmed for the primary key of the display function. For example, there is an update data for changing the display page size and an update data for changing the display page color, then the number of data rows of the target update data for the primary key of the display function is confirmed to be 2.

[0103] In a possible implementation, Figure 3As shown, the present application provides a data lake updating device based on artificial intelligence, the device comprising:

[0104] The data acquisition module 201 is used to obtain target data, where the target data includes update data and primary key data targeted by the update data, and the target data is used to update the data lake;

[0105] A merging module 202 is used to identify the number of data rows of target update data for a target primary key in the update data, and if the number of data rows is greater than or equal to a preset number of rows, merge the target update data to obtain merged data, and then obtain a merge event time of a merge event corresponding to the merged data;

[0106] An extraction module 203 is used to extract historical data of the target primary key in the data lake to be updated, and obtain the historical event time of the historical event corresponding to the historical data;

[0107] A comparison module 204 is used to compare the combined event time with the historical event time to obtain a comparison result, and determine the benchmark data and supplementary data according to the comparison result;

[0108] The updating module 205 is used to complete the update of the to-be-updated data lake according to the benchmark data and the supplementary data.

[0109] In a possible implementation, the merging module 202 is configured to:

[0110] Identify the number of data rows of target update data for a target primary key in the update data, and if the number of data rows is greater than or equal to a preset number of rows, obtain a target update event corresponding to each target update data, and then obtain a target update event time of the target update event according to the target update event;

[0111] Arrange each target update data according to the target update event time to obtain a data queue;

[0112] The merged data is obtained based on the data queue, and then the merged event time of the merged event corresponding to the merged data is obtained.

[0113] In a possible implementation, the merging module 202 is configured to:

[0114] Using the second target update data in the data queue as the reference data, using the first target update data in the data queue as the supplementary data, and using the remaining target update data in the data queue as the candidate target update data;

[0115] Determine whether the benchmark data has any default by traversing the benchmark data, and if the benchmark data has any default, obtain the positioning data of the default position on the benchmark data;

[0116] Extracting target data from the supplementary data based on the positioning data, and merging the target data with the reference data to obtain first merged data;

[0117] Merged data is obtained according to the first merged data and the candidate target update data, and then a merge event time of a merge event corresponding to the merged data is obtained.

[0118] In a possible implementation, the merging module 202 is configured to:

[0119] Using the first combined data as supplementary data and the candidate target updated data as reference data;

[0120] Determine whether the benchmark data has any default by traversing the benchmark data, and if the benchmark data has any default, obtain the positioning data of the default position on the benchmark data;

[0121] Target data is extracted from the supplementary data based on the positioning data, and the target data is merged with the reference data to obtain merged data, and then a merged event time of a merged event corresponding to the merged data is obtained.

[0122] In a possible implementation, the merging module 202 is configured to:

[0123] If the number of data rows is less than the preset number of rows, the target updated data is used as the merged data, and the step of obtaining the merged event time of the merged event corresponding to the merged data is continued.

[0124] In a possible implementation, the comparison module 204 is configured to:

[0125] Comparing the merged event time with the historical event time to obtain a comparison result, the comparison result comprising a first result indicating that the merged event time is greater than the historical event time, a second result indicating that the merged event time is less than the historical event time, and a third result indicating that the merged event time is equal to the historical event time;

[0126] When the comparison result is the first result, the historical data is used as supplementary data and the combined data is used as reference data;

[0127] When the comparison result is the second result, the historical data is used as the reference data, and the combined data is used as the supplementary data;

[0128] When the comparison result is the third result, the historical data is used as supplementary data, and the combined data is used as reference data.

[0129] In a possible implementation, the merging module 202 is configured to:

[0130] Randomly select any primary key data from the primary key data targeted by the update data as the target primary key data, and use the primary key corresponding to the target primary key data as the target primary key;

[0131] The target primary key data is matched with the primary key data targeted by the update data, and the number of data rows of the target update data targeted at the target primary key in the update data is identified according to the matching result.

[0132] The present invention provides a data lake update device, which obtains target data, including update data and primary key data targeted by the update data; identifies the number of data rows of the target update data for the target primary key in the update data, and if the number of data rows is greater than or equal to the preset number of rows, merges the target update data to obtain merged data, and then obtains the merge event time of the merge event corresponding to the merged data; extracts the historical data of the target primary key in the data lake to be updated, and obtains the historical event time of the historical event corresponding to the historical data; compares the merge event time with the historical event time to obtain a comparison result, and determines the benchmark data and supplementary data according to the comparison result; completes the update of the data lake to be updated according to the benchmark data and the supplementary data, and can support non-full field data not limited to OGG type entering the lake, which greatly expands the scope of Hudi use scenarios. And sorting according to event time, without relying on Flink watermark to sacrifice timeliness, solves the problem of data reverse order. After implementing non-full field entry into the lake, the relational database extracts CDC logs no longer need to extract data in a full field manner, effectively reducing the network bandwidth of data from the relational database to Kafka. It also reduces the storage capacity of CDC logs in Kafka, and reduces Kafka disk and memory consumption. That is, when programs and applications related to the smart medical or financial fields are updated, there is no need to call the complete installation package of the program or application for transmission, which reduces the computer resources occupied by the programs and applications related to the smart medical or financial fields when they are updated, and also reduces the update time of the programs and applications related to the smart medical or financial fields.

[0133] For the specific definition of the data lake update device, please refer to the definition of the data lake update method above, which will not be repeated here. Each module in the above-mentioned data lake update device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0134] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 4 As shown. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external client through a network connection. When the computer program is executed by the processor, it realizes the functions or steps of a service-side method for updating a data lake based on artificial intelligence.

[0135] In one embodiment, a computer device is provided. The computer device may be a client, and its internal structure diagram may be as follows: Figure 5 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected via a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external server via a network connection. When the computer program is executed by the processor, it implements the functions or steps of the client side of a data lake update method based on artificial intelligence.

[0136] In a possible implementation, Figure 6As shown, an embodiment of the present application provides an electronic device 300, including: a memory 310, a processor 320, and a computer program 311 stored in the memory 310 and executable on the processor 320. When the processor 320 executes the computer program 311, it implements: obtaining target data, wherein the target data includes update data and primary key data targeted by the update data; identifying the number of data rows of target update data for the target primary key in the update data, and if the number of data rows is greater than or equal to the preset number of rows, merging the target update data to obtain merged data, and then obtaining a merged event time of a merged event corresponding to the merged data; extracting historical data of the target primary key in the data lake to be updated, and obtaining the historical event time of the historical event corresponding to the historical data; comparing the merged event time with the historical event time to obtain a comparison result, and determining the baseline data and supplementary data according to the comparison result; completing the update of the data lake to be updated according to the baseline data and the supplementary data.

[0137] By acquiring the target data, the target data includes the updated data and the primary key data targeted by the updated data; identifying the number of data rows of the target updated data for the target primary key in the updated data, if the number of data rows is greater than or equal to the preset number of rows, the target updated data is merged to obtain the merged data, and then the merged event time of the merged event corresponding to the merged data is obtained; extracting the historical data of the target primary key in the data lake to be updated, and obtaining the historical event time of the historical event corresponding to the historical data; comparing the merged event time with the historical event time to obtain the comparison result, and determining the benchmark data and supplementary data according to the comparison result; completing the update of the data lake to be updated according to the benchmark data and the supplementary data, it can support non-full-field data not limited to OGG type entering the lake, which greatly expands the scope of Hudi use scenarios. And sorting according to event time, without relying on Flink watermark to sacrifice timeliness, solves the problem of data reverse order. After implementing non-full-field entry into the lake, the relational database extracts CDC logs no longer need to extract data in a full-field manner, effectively reducing the network bandwidth of data from the relational database to Kafka. It also reduces the storage capacity of CDC logs in Kafka, and reduces Kafka disk and memory consumption. That is, when programs and applications related to the smart medical or financial fields are updated, there is no need to call the complete installation package of the program or application for transmission, which reduces the computer resources occupied by the programs and applications related to the smart medical or financial fields when they are updated, and also reduces the update time of the programs and applications related to the smart medical or financial fields.

[0138] In a possible implementation, Figure 7As shown, an embodiment of the present application provides a computer-readable storage medium 400, on which a computer program 411 is stored. When the computer program 411 is executed by a processor, it implements: obtaining target data, the target data including update data and primary key data targeted by the update data; identifying the number of data rows of target update data for the target primary key in the update data, and if the number of data rows is greater than or equal to the preset number of rows, merging the target update data to obtain merged data, and then obtaining a merged event time of a merged event corresponding to the merged data; extracting historical data of the target primary key in the data lake to be updated, and obtaining the historical event time of the historical event corresponding to the historical data; comparing the merged event time with the historical event time to obtain a comparison result, and determining baseline data and supplementary data based on the comparison result; and completing the update of the data lake to be updated based on the baseline data and the supplementary data.

[0139] By acquiring the target data, the target data includes the updated data and the primary key data targeted by the updated data; identifying the number of data rows of the target updated data for the target primary key in the updated data, if the number of data rows is greater than or equal to the preset number of rows, the target updated data is merged to obtain the merged data, and then the merged event time of the merged event corresponding to the merged data is obtained; extracting the historical data of the target primary key in the data lake to be updated, and obtaining the historical event time of the historical event corresponding to the historical data; comparing the merged event time with the historical event time to obtain the comparison result, and determining the benchmark data and supplementary data according to the comparison result; completing the update of the data lake to be updated according to the benchmark data and the supplementary data, it can support non-full-field data not limited to OGG type entering the lake, which greatly expands the scope of Hudi use scenarios. And sorting according to event time, without relying on Flink watermark to sacrifice timeliness, solves the problem of data reverse order. After implementing non-full-field entry into the lake, the relational database extracts CDC logs no longer need to extract data in a full-field manner, effectively reducing the network bandwidth of data from the relational database to Kafka. It also reduces the storage capacity of CDC logs in Kafka, and reduces Kafka disk and memory consumption. That is, when programs and applications related to the smart medical or financial fields are updated, there is no need to call the complete installation package of the program or application for transmission, which reduces the computer resources occupied by the programs and applications related to the smart medical or financial fields when they are updated, and also reduces the update time of the programs and applications related to the smart medical or financial fields.

[0140] The computer storage medium of the embodiment of the present invention can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to: an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, a computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it.

[0141] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0142] The program code embodied on the computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0143] Computer program code for performing the operations of the present invention may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0144] It should be understood by those skilled in the art that the modules or steps of the present invention described above can be implemented by a general-purpose computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, optionally, they can be implemented by a program code executable by a computer device, so that they can be stored in a storage device and executed by the computing device, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0145] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described above, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

[0146] The above disclosure is only the preferred embodiment of the present invention, which certainly cannot be used to limit the scope of the present invention. Therefore, equivalent changes made according to the claims of the present invention are still within the scope of the present invention.

Claims

1. A data lake update method based on artificial intelligence, characterized in that: include: Obtain target data, where the target data includes update data and primary key data targeted by the update data, and the target data is used for updating the data lake; Identify the number of data rows of target update data for a target primary key in the update data, and if the number of data rows is greater than or equal to a preset number of rows, merge the target update data to obtain merged data, and then obtain a merge event time of a merge event corresponding to the merged data; Extracting historical data of the target primary key in the data lake to be updated, and obtaining historical event time of historical events corresponding to the historical data; Comparing the combined event time with the historical event time to obtain a comparison result, and determining the baseline data and the supplementary data according to the comparison result; The update of the data lake to be updated is completed according to the benchmark data and the supplementary data.

2. The data lake updating method based on artificial intelligence according to claim 1, characterized in that: The step of identifying the number of data rows of target update data for a target primary key in the update data, and if the number of data rows is greater than or equal to a preset number of rows, merging the target update data to obtain merged data, and then obtaining a merged event time of a merged event corresponding to the merged data, comprises: Identify the number of data rows of target update data for a target primary key in the update data, and if the number of data rows is greater than or equal to a preset number of rows, obtain a target update event corresponding to each target update data, and then obtain a target update event time of the target update event according to the target update event; Arrange each target update data according to the target update event time to obtain a data queue; The merged data is obtained based on the data queue, and then the merged event time of the merged event corresponding to the merged data is obtained.

3. The data lake updating method based on artificial intelligence according to claim 2, characterized in that: The step of obtaining the merged data based on the data queue, and then obtaining the merged event time of the merged event corresponding to the merged data, includes: Using the second target update data in the data queue as the reference data, using the first target update data in the data queue as the supplementary data, and using the remaining target update data in the data queue as the candidate target update data; Determine whether the benchmark data has any default by traversing the benchmark data, and if the benchmark data has any default, obtain the positioning data of the default position on the benchmark data; Extracting target data from the supplementary data based on the positioning data, and merging the target data with the reference data to obtain first merged data; Merged data is obtained according to the first merged data and the candidate target update data, and then a merged event time of a merged event corresponding to the merged data is obtained.

4. The data lake updating method based on artificial intelligence as claimed in claim 3, characterized in that: The step of obtaining merged data according to the first merged data and the candidate target update data, and then acquiring the merged event time of the merged event corresponding to the merged data, comprises: Using the first combined data as supplementary data and the candidate target updated data as reference data; Determine whether the benchmark data has any default by traversing the benchmark data, and if the benchmark data has any default, obtain the positioning data of the default position on the benchmark data; Target data is extracted from the supplementary data based on the positioning data, and the target data is merged with the reference data to obtain merged data, and then a merged event time of a merged event corresponding to the merged data is obtained.

5. The data lake updating method based on artificial intelligence according to claim 1, characterized in that: After the step of identifying the number of data rows of the target update data for the target primary key in the update data, the method further includes: If the number of data rows is less than the preset number of rows, the target updated data is used as the merged data, and the step of obtaining the merged event time of the merged event corresponding to the merged data is continued.

6. The data lake updating method based on artificial intelligence according to claim 1, characterized in that: The step of comparing the combined event time with the historical event time to obtain a comparison result, and determining the benchmark data and the supplementary data according to the comparison result includes: Comparing the merged event time with the historical event time to obtain a comparison result, the comparison result comprising a first result indicating that the merged event time is greater than the historical event time, a second result indicating that the merged event time is less than the historical event time, and a third result indicating that the merged event time is equal to the historical event time; When the comparison result is the first result, the historical data is used as supplementary data and the combined data is used as reference data; When the comparison result is the second result, the historical data is used as the reference data, and the combined data is used as the supplementary data; When the comparison result is the third result, the historical data is used as supplementary data, and the combined data is used as reference data.

7. The data lake updating method based on artificial intelligence according to claim 1, characterized in that: The step of identifying the number of data rows of target update data for a target primary key in the update data includes: Randomly select any primary key data from the primary key data targeted by the update data as the target primary key data, and use the primary key corresponding to the target primary key data as the target primary key; The target primary key data is matched with the primary key data targeted by the update data, and the number of data rows of the target update data targeted at the target primary key in the update data is identified according to the matching result.

8. An artificial intelligence-based data lake updating device, characterized in that: The device comprises: A data acquisition module is used to obtain target data, where the target data includes update data and primary key data targeted by the update data, and the target data is used to update the data lake; a merging module, used to identify the number of data rows of target update data for a target primary key in the update data, and if the number of data rows is greater than or equal to a preset number of rows, merge the target update data to obtain merged data, and then obtain a merge event time of a merge event corresponding to the merged data; An extraction module, used to extract historical data of the target primary key in the data lake to be updated, and obtain the historical event time of the historical event corresponding to the historical data; A comparison module, used for comparing the combined event time with the historical event time to obtain a comparison result, and determining the benchmark data and the supplementary data according to the comparison result; An updating module is used to complete the update of the to-be-updated data lake according to the benchmark data and the supplementary data.

9. An electronic device, characterized in that: include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate through the bus, and when the machine-readable instructions are executed by the processor, the steps of the artificial intelligence-based data lake update method as described in any one of claims 1 to 7 are performed.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the artificial intelligence-based data lake updating method according to any one of claims 1 to 7 are executed.

Images