Lake entering data inspection method, system and equipment and storage medium

Abstract

The invention discloses a lake-entering data inspection method, system and device and a storage medium, and belongs to the technical field of big data, and the method comprises the steps: obtaining a lake-entering mode of to-be-entering data; according to the lake entering mode, in-place inspection is conducted on the to-be-entering-into-lake data, and a first inspection result is obtained; according to the first inspection result, performing consistency inspection on the to-be-accessed lake data to obtain a second inspection result; and feeding back the first inspection result and the second inspection result to the data lake entering platform, so that the data lake entering platform controls the execution of the lake batch task corresponding to the to-be-into-lake data according to the first inspection result or the second inspection result. According to the method and the device, the problem that the lake batch task corresponding to the to-be-accessed data fails to be executed due to the fact that the to-be-accessed data is not in place or the to-be-accessed data is incomplete is solved, so that the success rate of one-time running of the lake batch task in the data accessing platform is improved, and the processing efficiency of the data accessing platform on the lake batch task is improved.

Description

Technical Field

[0001] The present invention relates to the field of big data technology, and in particular to a method, system, device and storage medium for checking lake data. Background Art

[0002] At present, the data entry platform generally calls up the lake batch task for execution at a fixed time and point. During the execution of the lake batch task, if it is found that a certain lake entry data is not in place, or if it is found that the data of a certain lake entry data is incomplete, the execution will be interrupted and an alarm message will be sent. In this way, the success rate of the one-time operation of the lake batch task in the data entry platform is reduced, affecting the efficiency of the data entry platform in processing the lake batch task. Summary of the invention

[0003] The present invention provides a method, system, device and storage medium for checking data entering a lake, so as to improve the success rate of one-time running of batch tasks of the lake in a data entering the lake platform, and improve the processing efficiency of batch tasks of the lake by the data entering the lake platform.

[0004] According to one aspect of the present invention, a method for checking data entering a lake is provided, the method comprising:

[0005] The method of obtaining the data to be entered into the lake;

[0006] According to the way of entering the lake, the data to be entered into the lake is checked in place to obtain the first inspection result;

[0007] According to the first check result, the data to be entered into the lake is checked for consistency to obtain the second check result;

[0008] The first inspection result and the second inspection result are fed back to the data entry platform, so that the data entry platform controls the execution of the lake batch tasks corresponding to the data to be entered into the lake according to the first inspection result or the second inspection result.

[0009] According to another aspect of the present invention, a lake entry data exploration system is provided, which includes: a data acquisition module, a data in place check module, a data consistency check module, a check result feedback module, a sentiment analysis module, a task classification module and an alarm module; wherein,

[0010] The data acquisition module is used to obtain the data information of the data to be entered into the lake; the data information of the data to be entered into the lake includes the entry method, the name of the entry file, the time of batch call-up of the lake, the application required by the file, the application to which the source data belongs, and the registration data structure;

[0011] A data in-place check module is used to perform an in-place check on the data to be entered into the lake according to the entry mode, and obtain a first check result;

[0012] A data consistency check module is used to perform a consistency check on the data to be entered into the lake according to the first check result to obtain a second check result;

[0013] An inspection result feedback module is used to feed back the first inspection result and the second inspection result to the data entry platform, so that the data entry platform controls the execution of the lake batch task corresponding to the data to be entered into the lake according to the first inspection result or the second inspection result;

[0014] The sentiment analysis module is used to perform sentiment analysis on the problem lake batch tasks in the data inflow platform to obtain the problem labels corresponding to the problem lake batch tasks;

[0015] The task classification module is used to classify the problem lake batch tasks in the data entry platform according to the problem labels corresponding to each problem lake batch task in the data entry platform, and obtain at least one task group so that relevant staff can carry out special governance for each obtained task group;

[0016] The alarm module is used to send an alarm message according to the first inspection result or the second inspection result.

[0017] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0018] at least one processor; and

[0019] a memory communicatively connected to at least one processor; wherein,

[0020] The memory stores a computer program that can be executed by at least one processor, and the computer program is executed by at least one processor so that the at least one processor can execute the lake entry data inspection method of any embodiment of the present invention.

[0021] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to implement the lake entry data inspection method of any embodiment of the present invention when executed by a processor.

[0022] According to another aspect of the present invention, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the method for checking lake entry data according to any embodiment of the present invention is implemented.

[0023] The technical solution of the embodiment of the present invention is to obtain the entry method of the data to be entered into the lake; according to the entry method, perform a presence check on the data to be entered into the lake to obtain a first check result; according to the first check result, perform a consistency check on the data to be entered into the lake to obtain a second check result; and feed back the first check result and the second check result to the data entry platform, so that the data entry platform controls the execution of the lake batch tasks corresponding to the data to be entered into the lake according to the first check result or the second check result. The above technical scheme, before the data entry platform executes the lake batch tasks corresponding to the data to be entered into the lake, first performs a check on the data to be entered into the lake, and after the data to be entered into the lake passes the check on the availability (i.e., determines that the first check result is that the data is in place), performs a consistency check on the data to be entered into the lake, and after the data to be entered into the lake passes the consistency check (i.e., determines that the second check result is that the data is consistent), notifies the data entry platform to execute the lake batch tasks corresponding to the data to be entered into the lake, which solves the problem of failure to execute the lake batch tasks corresponding to the data to be entered into the lake due to the data to be entered into the lake not being in place or the data to be entered into the lake being incomplete, thereby improving the success rate of one-time running of the lake batch tasks in the data entry platform, and improving the processing efficiency of the data entry platform for the lake batch tasks.

[0024] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0026] Figure 1 is a flow chart of a method for checking lake entry data provided according to Embodiment 1 of the present invention;

[0027] Figure 2 is a flow chart of a method for checking lake entry data provided according to Embodiment 2 of the present invention;

[0028] Figure 3 is a structural diagram of a lake entry data exploration system provided according to Embodiment 3 of the present invention;

[0029] Figure 4 It is a structural schematic diagram of an electronic device for implementing the lake entry data inspection method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0030] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0031] It should be noted that the terms "first" and "second" and the like in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0032] In addition, it should be noted that in the technical solution of the present invention, the collection, storage, use, processing, transmission, provision and disclosure of problem lake batch tasks in the data entry platform are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0033] Embodiment 1

[0034] Figure 1 This is a flow chart of a method for checking lake data provided in the first embodiment of the present invention. This embodiment can be applied to the case where the lake data in the lake batch task is checked before the data lake entry platform executes the lake batch task. The method is implemented by means of a lake data exploration system, which can be configured in an electronic device. Figure 1 As shown, the method includes:

[0035] S101, obtaining a method for entering the lake for the data to be entered into the lake.

[0036] Among them, the data to be entered into the lake refers to the data to be entered into the lake in the batch tasks to be executed in the data entry platform. Among them, the data entry platform refers to a platform used to integrate data from various sources, forms and types into the data lake. The data lake is a large-scale data storage and management system that can contain structured data, semi-structured data and unstructured data. The entry method into the lake refers to the way in which the data to be entered into the lake enters the data lake; optionally, the entry method into the lake includes but is not limited to the point-to-point file transfer method, the file splitting platform acquisition method and the high-efficiency entry method into the lake. It should be noted that the point-to-point file transfer method and the file splitting platform acquisition method refer to the data to be entered into the lake entering the data lake in the form of files. The high-efficiency entry method into the lake refers to the data to be entered into the lake entering the data lake in the form of storage in a temporary data table.

[0037] Specifically, the entry method of the data to be entered into the lake can be obtained from the entry data registration form in the data entry platform. The entry data registration form refers to a data table used to register the data information of the data to be entered into the lake.

[0038] S102. According to the method of entering the lake, the data to be entered into the lake is checked to obtain a first inspection result.

[0039] The first inspection result refers to the result obtained after the data to be entered into the lake has been checked for being in place; optionally, the first inspection result may be that the data is in place or the data is not in place.

[0040] Specifically, according to the lake entry method, based on the correspondence between the candidate lake entry methods and the candidate in-place inspection rules in the preset database, the target in-place inspection rule corresponding to the lake entry method can be determined from the preset database; the target in-place inspection rule is used to perform an in-place inspection on the data to be entered into the lake to obtain a first inspection result.

[0041] Among them, the preset database may refer to a database pre-configured in the lake entry data exploration system, which is used to store pre-configured data entry methods and data in place check rules. Among them, the candidate lake entry method refers to the lake entry method stored in the preset database. The candidate in place check rule refers to the data in place check rule stored in the preset database. The target in place check rule refers to the candidate lake entry method in the preset database that corresponds to the lake entry method of the data to be entered into the lake.

[0042] S103. Based on the first check result, a consistency check is performed on the data to be entered into the lake to obtain a second check result.

[0043] Specifically, when the first check result is that the data is in place, the registration data structure of the data to be entered into the lake can be obtained; it can be checked whether the current data structure of the data to be entered into the lake is consistent with the registration data structure; if so, it is determined that the second check result is that the data is consistent; otherwise, it is determined that the second check result is that the data is inconsistent.

[0044] The registered data structure refers to the data structure of the data to be entered into the lake, and the current data structure refers to the data structure of the data to be entered into the lake at the current moment.

[0045] More specifically, when the first check result is that the data is in place, that is, when the data to be entered into the lake passes the in place check, if the data to be entered into the lake is entered into the lake in a point-to-point file transfer method or a file splitting platform acquisition method, then the name of the entry file of the data to be entered into the lake is used as an index to obtain the registration data structure of the data to be entered into the lake from the registration information table of the entry file; if the entry method of the data to be entered into the lake is a high-efficiency entry method, then the name of the entry file of the data to be entered into the lake is used as an index to obtain the registration data structure of the data to be entered into the lake from the source table registration information; thereafter, it is detected whether the current data structure of the data to be entered into the lake is consistent with the registration data structure; if so, it is determined that the second check result is that the data is consistent; otherwise, it is determined that the second check result is that the data is inconsistent.

[0046] Among them, the registration information table for files entering the lake refers to a data table used to register relevant file information of files entering the lake; wherein the relevant file information includes but is not limited to the data structure, file format, file size and number of files entering the lake. It should be noted that the registration information table for files entering the lake is stored in the data source platform for the data to be entered into the lake. Among them, the data source platform refers to the platform that provides the data to be entered into the lake; optionally, the data source platform can be a resource information platform, a file transfer platform, a file splitting platform or a big data platform.

[0047] The source table registration information refers to the information generated when registering and recording the data table that provides the data to be entered into the lake. It should be noted that the source table registration information is stored in the data source platform of the data to be entered into the lake. The data source platform can be a resource information platform or a big data platform.

[0048] It should also be noted that when the first check result is that the data is not in place, the data to be entered into the lake will not be checked for consistency. Instead, the first check result will be fed back to the data entry platform so that the data entry platform can postpone the execution of the lake batch task corresponding to the data to be entered into the lake based on the first check result.

[0049] It is understandable that after the data to be entered into the lake passes the arrival check, data consistency check is continued on the data to be entered into the lake. While ensuring that the data to be entered into the lake is in place, the data integrity of the data to be entered into the lake is further ensured, and the problem of failure of execution of lake batch tasks corresponding to the data to be entered into the lake due to the arrival of the data to be entered into the lake or the incomplete data of the data to be entered into the lake is solved, thereby improving the success rate of one-time running of lake batch tasks in the data entry platform, and improving the processing efficiency of the data entry platform for lake batch tasks.

[0050] Optionally, after determining that the second check result is inconsistent data, an alarm message is sent according to the lake entry method, and a task postponement instruction is sent to the data lake entry platform, so that the data lake entry platform postpones the execution of the lake batch task corresponding to the data to be entered into the lake according to the task postponement instruction. The task postponement instruction refers to an instruction used to instruct the data lake entry platform to postpone the execution of the lake batch task.

[0051] Specifically, after determining that the second check result is data inconsistency, that is, after the data to be entered into the lake fails the consistency check, if the entry method of the data to be entered into the lake is a point-to-point file transfer method, a file error alarm message is sent to the application to which the source file corresponding to the data to be entered into the lake belongs, and a task postponement instruction is sent to the data entry platform, so that the data entry platform postpones the execution of the lake batch task corresponding to the data to be entered into the lake according to the task postponement instruction; if the entry method of the data to be entered into the lake is a file splitting platform acquisition method, a file error alarm message is sent to the application to which the file before splitting corresponding to the data to be entered into the lake belongs, and a task postponement instruction is sent to the data entry platform, so that the data entry platform postpones the execution of the lake batch task corresponding to the data to be entered into the lake according to the task postponement instruction; if the entry method of the data to be entered into the lake is a high-efficiency entry method, a data error alarm message is sent to the application to which the source data corresponding to the data to be entered into the lake belongs, and a task postponement instruction is sent to the data entry platform, so that the data entry platform postpones the execution of the lake batch task corresponding to the data to be entered into the lake according to the task postponement instruction.

[0052] The source file belongs to the application that provides the data to be entered into the lake. The pre-splitting file belongs to the application that the file that provides the data to be entered into the lake belongs to before the split. The source data belongs to the application that provides the data source of the data to be entered into the lake.

[0053] It is understandable that when the data to be entered into the lake passes the in-place check but fails the consistency check, an alarm message is sent to the corresponding application in a targeted manner according to the entry method of the data to be entered into the lake, so as to remind the corresponding application to promptly discover and correct the data structure of the data to be entered into the lake. At the same time, a task postponement instruction is sent to the data entry platform, so that the data entry platform postpones the execution of the lake batch task corresponding to the data to be entered into the lake according to the task postponement instruction, thereby avoiding the failure of the execution of the lake batch task corresponding to the data to be entered into the lake due to incomplete data of the data to be entered into the lake, thereby improving the success rate of the one-time operation of the lake batch task in the data entry platform, and improving the processing efficiency of the lake batch task by the data entry platform.

[0054] Optionally, after determining that the second check result is inconsistent data, while sending an alarm message according to the entry method, relevant error information of the data to be entered into the lake can also be recorded according to the entry method. Specifically, if the entry method of the data to be entered into the lake is a point-to-point file transfer method, then the relevant error information such as the name of the entry file of the data to be entered into the lake, the application name to which the source file belongs, the lake batch task name, and the priority corresponding to the lake batch task are recorded; if the entry method of the data to be entered into the lake is a file splitting platform acquisition method, then the name of the entry file of the data to be entered into the lake, the number of files entering the lake, the file name before splitting, the application name to which the file belongs before splitting, the lake batch task name, and the priority corresponding to the lake batch task are recorded; if the entry method of the data to be entered into the lake is a high-efficiency entry method, then the relevant error information such as the application name to which the source data to be entered into the lake belongs, the lake batch task name, the task processing algorithm corresponding to the lake batch task, and the priority corresponding to the lake batch task are recorded, so as to trace the cause of the alarm later.

[0055] S104: Feedback the first inspection result and the second inspection result to the data entry platform, so that the data entry platform controls the execution of the lake batch task corresponding to the data to be entered into the lake according to the first inspection result or the second inspection result.

[0056] Specifically, the first inspection result and the second inspection result are fed back to the data entry platform. If the first inspection result is that the data is not in place, the data entry platform will postpone the execution of the lake batch task corresponding to the data to be entered into the lake based on the first inspection result; if the first inspection result is that the data is in place, but the second inspection result is that the data is inconsistent, the data entry platform will postpone the execution of the lake batch task corresponding to the data to be entered into the lake based on the second inspection result; if the first inspection result is that the data is in place and the second inspection result is that the data is consistent, the data entry platform will normally execute the lake batch task corresponding to the data to be entered into the lake based on the first inspection result and the second inspection result.

[0057] The technical solution of the embodiment of the present invention is to obtain the entry method of the data to be entered into the lake; according to the entry method, perform a presence check on the data to be entered into the lake to obtain a first check result; according to the first check result, perform a consistency check on the data to be entered into the lake to obtain a second check result; and feed back the first check result and the second check result to the data entry platform, so that the data entry platform controls the execution of the lake batch tasks corresponding to the data to be entered into the lake according to the first check result or the second check result. The above technical scheme, before the data entry platform executes the lake batch tasks corresponding to the data to be entered into the lake, first performs a check on the data to be entered into the lake, and after the data to be entered into the lake passes the check on the availability (i.e., determines that the first check result is that the data is in place), performs a consistency check on the data to be entered into the lake, and after the data to be entered into the lake passes the consistency check (i.e., determines that the second check result is that the data is consistent), notifies the data entry platform to execute the lake batch tasks corresponding to the data to be entered into the lake, which solves the problem of failure to execute the lake batch tasks corresponding to the data to be entered into the lake due to the data to be entered into the lake not being in place or the data to be entered into the lake being incomplete, thereby improving the success rate of one-time running of the lake batch tasks in the data entry platform, and improving the processing efficiency of the data entry platform for the lake batch tasks.

[0058] Embodiment 2

[0059] Figure 2 This is a flow chart of a method for checking data entering the lake provided in Example 2 of the present invention. Based on the above example, this example further optimizes "according to the method of entering the lake, the data to be entered into the lake is checked in place to obtain a first inspection result" and provides an optional implementation plan. It should be noted that for the parts not described in detail in the example of the present invention, reference can be made to the relevant descriptions of other examples. Figure 2 As shown, the method includes:

[0060] S201, obtaining a method for entering the lake for the data to be entered into the lake.

[0061] Specifically, the entry method of the data to be entered into the lake can be obtained through the data acquisition module in the lake entry data exploration system.

[0062] S202: If the method of entering the lake is a point-to-point file transfer method or a file splitting platform acquisition method, obtain the lake entry file name of the data to be entered into the lake and the lake batch call-up time corresponding to the data to be entered into the lake.

[0063] The lake entry file name refers to the file name when the data to be entered into the lake enters the data lake in the form of a file. The lake batch call time refers to the time when the data entry platform executes the lake batch task corresponding to the data to be entered into the lake.

[0064] Specifically, if the method of entering the lake is a point-to-point file transfer method or a file splitting platform acquisition method, the data acquisition module in the lake entry data exploration system can be used to obtain the lake entry file name and the lake batch call-up time corresponding to the data to be entered into the lake.

[0065] S203: Determine the target storage location of the data to be stored in the lake according to the name of the file to be stored in the lake.

[0066] Among them, the target storage location refers to the storage location where the data to be entered into the lake is located before entering the lake.

[0067] Specifically, the name of the data entry file can be used as an index to determine the target storage location of the data to be entered into the lake from the designated storage space of the data entry platform.

[0068] S204: Detect whether the data to be entered into the lake arrives at the target storage location before the lake batch call time.

[0069] Specifically, before the lake batch call time, it is detected whether there is data to be entered into the lake at the target storage location; if so, S205 is executed; if not, S206 is executed.

[0070] S205: Determine that the first inspection result is that the data is in place.

[0071] Specifically, before the lake batch call time, if it is detected that there is data to be entered into the lake at the target storage location, it is determined that the first check result is that the data is in place. Then, S207 is executed.

[0072] S206: Determine that the first inspection result is that data is not in place.

[0073] Specifically, before the lake batch call time, if it is detected that there is no data to be entered into the lake at the target storage location, it is determined that the first check result is that the data is not in place. Then, S208 is executed.

[0074] Optionally, after determining that the first check result is that the data is not in place, an alarm message can be sent to the application required by the file corresponding to the data to be entered into the lake; and a task postponement instruction can be sent to the data entry platform, so that the data entry platform can postpone the execution of the lake batch task corresponding to the data to be entered into the lake according to the task postponement instruction.

[0075] Among them, the application required for the file refers to the application that needs the data to be entered into the lake. Specifically, after determining that the first inspection result is that the data is not in place, that is, after determining that the data to be entered into the lake by the point-to-point file transfer method or the file splitting platform acquisition method has not passed the in-place inspection, an alarm message is sent to the application required for the file corresponding to the data to be entered into the lake to inform the application required for the file that the data to be entered into the lake has not arrived and needs to be waited; at the same time, a task postponement instruction is sent to the data entry platform, so that the data entry platform postpones the execution of the lake batch task corresponding to the data to be entered into the lake according to the task postponement instruction, which solves the problem of failure to execute the lake batch task corresponding to the data to be entered into the lake due to the in-place data to be entered into the lake.

[0076] Optionally, after determining that the first inspection result is that the data is not in place, relevant alarm information of the data to be entered into the lake can also be recorded according to the entry method. Specifically, if the entry method of the data to be entered into the lake is a point-to-point file transfer method, then the waiting time required for the application of the file to be entered into the lake, the name of the entry file, the name of the application to which the source file belongs, the IP address of the server where the source file is located, the lake batch task name and the priority corresponding to the lake batch task and other related alarm information are recorded; if the entry method of the data to be entered into the lake is a file splitting platform acquisition method, then the waiting time required for the application of the file to be entered into the lake, the name of the entry file, the name of the file before splitting, the name of the application to which the file belongs before splitting, the lake batch task name and the priority corresponding to the lake batch task and other related alarm information are recorded, so as to trace the cause of the alarm later.

[0077] S207. When it is determined that the first check result is that the data is in place, a consistency check is performed on the data to be entered into the lake to obtain a second check result.

[0078] S208. Feedback the first inspection result and the second inspection result to the data entry platform, so that the data entry platform controls the execution of the lake batch task corresponding to the data to be entered into the lake according to the first inspection result or the second inspection result.

[0079] The technical solution of the embodiment of the present invention is to obtain the entry mode of the data to be entered into the lake; if the entry mode is a point-to-point file transfer mode or a file splitting platform acquisition mode, then obtain the entry file name of the data to be entered into the lake and the lake batch call-up time corresponding to the data to be entered into the lake; determine the target storage location of the data to be entered into the lake according to the entry file name; detect whether the data to be entered into the lake arrives at the target storage location before the lake batch call-up time; if so, determine that the first check result is that the data is in place; otherwise, determine that the first check result is that the data is not in place; in the case where it is determined that the first check result is that the data is in place, perform a consistency check on the data to be entered into the lake to obtain a second check result; feed back the first check result and the second check result to the data entry platform, so that the data entry platform controls the execution of the lake batch task corresponding to the data to be entered into the lake according to the first check result or the second check result. The above technical solution, before the data entry platform executes the lake batch task corresponding to the data to be entered into the lake, firstly performs a targeted in-place check on the data to be entered into the lake according to the entry mode of the data to be entered into the lake to obtain the first check result, so that the in-place check of the data to be entered into the lake is more in line with the actual situation, thereby improving the accuracy of the first check result. Afterwards, when the first check result is that the data is in place, a consistency check is performed on the data to be entered into the lake to obtain a second check result, and the second check result is fed back to the data entry platform, so that the data entry platform can control the execution of the lake batch tasks corresponding to the data to be entered into the lake according to the second check result; when the first check result is that the data is not in place, the first check result is fed back to the data entry platform, so that the data entry platform postpones the execution of the lake batch tasks corresponding to the data to be entered into the lake according to the first check result, which solves the problem of failure to execute the lake batch tasks corresponding to the data to be entered into the lake due to the non-availability of the data to be entered into the lake or the incomplete data of the data to be entered into the lake, thereby improving the success rate of one-time running of the lake batch tasks in the data entry platform, and improving the processing efficiency of the data entry platform for the lake batch tasks.

[0080] On the basis of the above embodiments, as an optional method of the embodiments of the present invention, according to the lake entry method, the data to be entered into the lake is checked for in place to obtain a first inspection result. It can also be: if the lake entry method is a high-efficiency lake entry method, the temporary data table and the lake batch call-up time corresponding to the data to be entered into the lake are obtained; it is detected whether the temporary data table arrives at the data lake entry platform before the lake batch call-up time; if so, it is determined that the first inspection result is that the data is in place; otherwise, it is determined that the first inspection result is that the data is not in place.

[0081] Among them, the temporary data table refers to a data table used to temporarily store the data to be entered into the lake. Specifically, if the entry into the lake is a high-efficiency entry into the lake method, the temporary data table corresponding to the data to be entered into the lake can be obtained from the big data platform through the data acquisition module in the lake entry data exploration system, and the lake batch call-up time corresponding to the data to be entered into the lake can be obtained from the data entry platform; before the lake batch call-up time, check whether there is a temporary data table corresponding to the data to be entered into the lake in the data entry platform; if so, determine that the first check result is that the data is in place; if not, determine that the first check result is that the data is not in place.

[0082] It can be understood that the targeted on-site inspection of the data to be entered into the lake with high timeliness makes the on-site inspection of the data to be entered into the lake more in line with the actual situation, thereby improving the accuracy of the first inspection result.

[0083] Optionally, after determining that the first check result is that the data is not in place, check whether there is data to be entered into the lake in the Kafka message queue; if so, send an alarm message to the distributed message platform to which the Kafka message queue belongs; otherwise, send an alarm message to the application to which the source data corresponding to the data to be entered into the lake belongs.

[0084] Specifically, after determining that the first check result is that the data is not in place, that is, after determining that the data to be entered into the lake in the high-efficiency entry mode has not passed the in-place check, check whether there is data to be entered into the lake in the Kafka message queue; if so, it indicates that there is data to be entered into the lake in the Kafka message queue, but the data to be entered into the lake has not yet been written to the temporary data table, and an alarm message is sent to the distributed message platform to which the Kafka message queue belongs, so as to remind the distributed message platform to write the data to be entered into the temporary data table as soon as possible; if not, it indicates that there is no data to be entered into the lake in the Kafka message queue, and the application to which the source data corresponding to the data to be entered into the lake has not used the data copy tool to copy the data to be entered into the lake to the Kafka message queue, and an alarm message is sent to the application to which the source data corresponding to the data to be entered into the lake belongs, so as to remind the application to which the source data corresponding to the data to be entered into the lake to copy the data to be entered into the lake to the Kafka message queue as soon as possible, so as to perform subsequent operations.

[0085] Embodiment 3

[0086] Figure 3 This is a schematic diagram of the structure of a lake entry data exploration system provided in the third embodiment of the present invention. This embodiment can be applied to the situation where the lake entry data in the lake batch task is checked before the data lake entry platform executes the lake batch task, and can also be applied to the situation where the problematic lake batch tasks in the data lake entry platform are classified and managed. The lake entry data exploration system can be configured in an electronic device. Figure 3As shown, the lake data exploration system 300 includes: a data acquisition module 301, a data in place check module 302, a data consistency check module 303, a check result feedback module 304, a sentiment analysis module 305, a task classification module 306 and an alarm module 307; wherein,

[0087] The data acquisition module 301 is used to acquire data information of the data to be entered into the lake; wherein the data information of the data to be entered into the lake includes the entry method, the entry file name, the lake batch call time, the application required by the file, the application to which the source data belongs, and the registration data structure;

[0088] A data in-place check module 302 is used to perform an in-place check on the data to be entered into the lake according to the entry mode, and obtain a first check result;

[0089] The data consistency check module 303 is used to perform a consistency check on the data to be entered into the lake according to the first check result to obtain a second check result;

[0090] The inspection result feedback module 304 is used to feed back the first inspection result and the second inspection result to the data entry platform, so that the data entry platform controls the execution of the lake batch task corresponding to the data to be entered into the lake according to the first inspection result or the second inspection result;

[0091] Sentiment analysis module 305, used to perform sentiment analysis on the problem lake batch tasks in the data entry platform to obtain the problem labels corresponding to the problem lake batch tasks;

[0092] The task classification module 306 is used to classify the problem lake batch tasks in the data entry platform according to the problem label corresponding to each problem lake batch task in the data entry platform, and obtain at least one task group so that relevant staff can carry out special treatment for each obtained task group;

[0093] The alarm module 307 is used to send an alarm message according to the first inspection result or the second inspection result.

[0094] It should be noted that the data acquisition module 301 is communicated with the data in place check module 302, the data in place check module 302 is communicated with the data consistency check module 303, the data in place check module 302 is communicated with the inspection result feedback module 304, the data consistency check module 303 is communicated with the inspection result feedback module 304, the data acquisition module 301 is communicated with the sentiment analysis module 305, the sentiment analysis module 305 is communicated with the task classification module 306, the data in place check module 302 is communicated with the alarm module 307, and the data consistency check module 303 is communicated with the alarm module 307.

[0095] Optionally, the data acquisition module 301 may send the acquired entry mode of the data to be entered into the lake to the data in place check module 302, so that the data in place check module 302 performs an in place check on the data to be entered into the lake according to the entry mode to obtain a first check result.

[0096] Optionally, the data in-place check module 302 may send the first check result obtained to the data consistency check module 303, the check result feedback module 304 and the alarm module 307, so that the data consistency check module 303 performs a consistency check on the data to be entered into the lake according to the first check result to obtain a second check result, so that the check result feedback module 304 feeds back the first check result to the data entry platform, so that the alarm module 307 sends an alarm message according to the first check result.

[0097] Optionally, the data consistency check module 303 may send the obtained second check result to the check result feedback module 304 and the alarm module 307, so that the check result feedback module 304 feeds back the second check result to the data entry platform, so that the alarm module 307 sends an alarm message according to the second check result.

[0098] Optionally, the data acquisition module 301 can send the problem lake batch tasks obtained from the data entry platform to the sentiment analysis module 305, so that the sentiment analysis module 305 performs sentiment analysis on the problem lake batch tasks in the data entry platform to obtain the problem labels corresponding to the problem lake batch tasks.

[0099] Optionally, the sentiment analysis module 305 can send the obtained problem lake batch tasks with problem labels to the task classification module 306, so that the task classification module 306 classifies the problem lake batch tasks in the data entry platform according to the problem labels corresponding to each problem lake batch task in the data entry platform to obtain at least one task group.

[0100] Among them, the sentiment analysis module 305 is composed of a sentiment analysis model, which is used to perform sentiment analysis on the problem lake batch tasks in the data entry platform. Among them, the sentiment analysis model can be obtained by training the machine learning model through a large number of sample lake batch tasks with problem labels. Problem lake batch tasks refer to lake batch tasks with problems; optionally, problem lake batch tasks can be lake batch tasks with the problem of inadequate data to be entered into the lake, or lake batch tasks with incomplete data to be entered into the lake. It should be noted that the relevant staff can be operation and maintenance personnel or developers.

[0101] For each problem lake batch task in the data entry platform, the sentiment analysis module 305 can analyze the problem label corresponding to the problem lake batch task. Exemplarily, for the problem lake batch task A in the data entry platform, the sentiment analysis module 305 performs sentiment analysis on the problem lake batch task A, and obtains the following problem label of the problem lake batch task A: the problem lake batch task A is delayed at xx / xx / xx / xx, the delay time is 20 minutes, the alarm type is file late, the name of the entry file is M, and the application to which the source file belongs is α.

[0102] It should be noted that the problem lake batch tasks with problem labels analyzed by the sentiment analysis module 305 can also be written into the training data set of the sentiment analysis model to enrich the training data of the sentiment analysis model, thereby continuously improving the accuracy of the sentiment analysis model, and further enabling the sentiment analysis module 305 to more accurately analyze the problem labels corresponding to the problem lake batch tasks in the data entry platform.

[0103] Among them, according to the problem label corresponding to each problem lake batch task in the data entry platform, the problem lake batch tasks in the data entry platform are classified to obtain at least one task group. Specifically, it can be: according to the alarm type in the problem label corresponding to the problem lake batch task in the data entry platform, the problem lake batch tasks in the data entry platform are classified to obtain at least one task group.

[0104] Exemplarily, if there are 5 problem lake batch tasks in the data entry platform, namely, problem lake batch task 1, problem lake batch task 2, problem lake batch task 3, problem lake batch task 4 and problem lake batch task; wherein, the alarm type in the problem tag corresponding to problem lake batch task 1 is file late arrival, the alarm type in the problem tag corresponding to problem lake batch task 2 is file data error, the alarm type in the problem tag corresponding to problem lake batch task 3 is file late arrival, the alarm type in the problem tag corresponding to problem lake batch task 4 is file data error, and the alarm type in the problem tag corresponding to problem lake batch task 5 is batch processing time too long. Based on the above information, the task classification module 306 in the lake entry data exploration system can divide the above 5 problem lake batch tasks into 3 task groups, namely, file late arrival task group, file data error task group and batch processing time too long task group; wherein, the file late arrival task group includes problem lake batch task 1 and problem lake batch task 3; the file data error task group includes problem lake batch task 2 and problem lake batch task 4; the batch processing time too long task group includes problem lake batch task 5.

[0105] It can be understood that according to the alarm type in the problem label corresponding to the problem lake batch task in the data entry platform, the problem lake batch tasks in the data entry platform can be classified through the task classification module 306 in the lake entry data exploration system. The problem lake batch tasks with the same alarm type can be divided into the same task group, so that relevant staff can carry out special management of the problem batch tasks with the same alarm type, thereby reducing manpower consumption.

[0106] Among them, the alarm module 307 can send an alarm message when the first check result is that the data is not in place, and the specific details can be referred to the records in Example 2; when the second check result is that the data is inconsistent, the alarm message can be sent, and the specific details can be referred to the records in Example 1.

[0107] The lake entry data exploration system provided by the embodiment of the present invention can, on the one hand, perform a check on the data to be entered into the lake before the data entry platform executes the lake batch tasks corresponding to the data to be entered into the lake, and after the data to be entered into the lake passes the check on the availability (i.e., determining that the first check result is that the data is in place), perform a consistency check on the data to be entered into the lake, and after the data to be entered into the lake passes the consistency check (i.e., determining that the second check result is that the data is consistent), notify the data entry platform to execute the lake batch tasks corresponding to the data to be entered into the lake, solving the problem of failure to execute the lake batch tasks corresponding to the data to be entered into the lake due to the incompleteness of the data to be entered into the lake or the incompleteness of the data to be entered into the lake, thereby improving the success rate of the one-time operation of the lake batch tasks in the data entry platform, and improving the processing efficiency of the lake batch tasks of the data entry platform. On the other hand, sentiment analysis and task classification of the problem lake batch tasks in the data entry platform can also be performed to achieve classified governance of the problem lake batch tasks in the data entry platform, thereby reducing manpower consumption.

[0108] According to an embodiment of the present invention, the present invention also provides an electronic device, a readable storage medium and a computer program product.

[0109] Embodiment 4

[0110] Figure 4 A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0111] like Figure 4 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, ROM 12 and RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0112] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0113] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as the lake entry data inspection method.

[0114] In some embodiments, the lake entry data inspection method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the lake entry data inspection method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to execute the lake entry data inspection method in any other appropriate manner (for example, by means of firmware).

[0115] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0116] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0117] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0118] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0119] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0120] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0121] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0122] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for checking lake entry data, characterized in that: include: The method of obtaining the data to be entered into the lake; According to the lake entry method, the data to be entered into the lake is checked for being in place, and a first inspection result is obtained; According to the first check result, a consistency check is performed on the data to be entered into the lake to obtain a second check result; The first inspection result and the second inspection result are fed back to the data entry platform, so that the data entry platform controls the execution of the lake batch task corresponding to the data to be entered into the lake according to the first inspection result or the second inspection result.

2. The method according to claim 1, characterized in that According to the lake entry method, the data to be entered into the lake is checked for being in place, and a first check result is obtained, including: If the entry mode is a point-to-point file transfer mode or a file splitting platform acquisition mode, then obtaining the entry file name of the data to be entered into the lake and the lake batch call-up time corresponding to the data to be entered into the lake; Determine the target storage location of the data to be entered into the lake according to the name of the file to be entered into the lake; Detecting whether the data to be entered into the lake arrives at the target storage location before the lake batch call-up time; If so, it is determined that the first check result is that the data is in place; Otherwise, it is determined that the first check result is that the data is not in place.

3. The method according to claim 1, characterized in that According to the lake entry method, the data to be entered into the lake is checked for being in place, and a first check result is obtained, including: If the lake entry method is a high-efficiency lake entry method, then obtain a temporary data table and lake batch call-up time corresponding to the data to be entered into the lake; Detect whether the temporary data table arrives at the data lake entry platform before the lake batch call time; If so, it is determined that the first check result is that the data is in place; Otherwise, it is determined that the first check result is that the data is not in place.

4. The method according to claim 2, characterized in that: After determining that the first check result is that the data is not in place, the method further includes: Send an alarm message to the application required by the file corresponding to the data to be entered into the lake; and send a task postponement instruction to the data entry platform, so that the data entry platform postpones the execution of the lake batch task corresponding to the data to be entered into the lake according to the task postponement instruction.

5. The method according to claim 3, characterized in that: After determining that the first check result is that the data is not in place, the method further includes: Check whether the data to be entered into the lake exists in the Kafka message queue; If yes, an alarm message is sent to the distributed message platform to which the Kafka message queue belongs; Otherwise, an alarm message is sent to the application to which the source data corresponding to the data to be entered into the lake belongs.

6. The method according to claim 1, characterized in that According to the first check result, the consistency check is performed on the data to be entered into the lake to obtain a second check result, including: When the first inspection result is that the data is in place, obtaining the registration data structure of the data to be entered into the lake; Detecting whether the current data structure of the data to be entered into the lake is consistent with the registered data structure; If yes, then the second check result is determined to be data consistent; Otherwise, it is determined that the second check result is data inconsistency.

7. The method according to claim 6, characterized in that After determining that the second check result is data inconsistency, the method further includes: According to the lake entry method, an alarm message is sent, and a task postponement instruction is sent to the data lake entry platform, so that the data lake entry platform postpones the execution of the lake batch task corresponding to the data to be entered into the lake according to the task postponement instruction.

8. A lake entry data exploration system, characterized in that: The lake data exploration system includes: a data acquisition module, a data in place check module, a data consistency check module, a check result feedback module, a sentiment analysis module, a task classification module and an alarm module; wherein, The data acquisition module is used to acquire data information of the data to be entered into the lake; wherein the data information of the data to be entered into the lake includes the entry method, the entry file name, the lake batch call time, the application required by the file, the application to which the source data belongs, and the registration data structure; The data in-place check module is used to perform an in-place check on the data to be entered into the lake according to the entry mode, and obtain a first check result; The data consistency check module is used to perform a consistency check on the data to be entered into the lake according to the first check result to obtain a second check result; The inspection result feedback module is used to feed back the first inspection result and the second inspection result to the data entry platform, so that the data entry platform controls the execution of the lake batch task corresponding to the data to be entered into the lake according to the first inspection result or the second inspection result; The sentiment analysis module is used to perform sentiment analysis on the problem lake batch tasks in the data entry platform to obtain the problem labels corresponding to the problem lake batch tasks; The task classification module is used to classify the problem lake batch tasks in the data entry platform according to the problem label corresponding to each problem lake batch task in the data entry platform, and obtain at least one task group so that relevant staff can carry out special governance for each obtained task group; The alarm module is used to send alarm information according to the first inspection result or the second inspection result.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to perform the lake entry data inspection method described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the lake entry data inspection method described in any one of claims 1-7 when executed.

Images