Automatic detection method, system and equipment for data development codes and storage medium

Abstract

The invention provides a data development code automatic detection method, system and device and a storage medium. The method comprises the steps of obtaining a to-be-detected data development code; judging whether the to-be-detected data development code is consistent with a code review rule or not, if not, modifying the to-be-detected data development code, and if yes, establishing a data blood relationship based on the to-be-detected data development code; and performing quality verification on the to-be-detected data development code, if the quality verification is not passed, positioning and modifying an error code based on the data blood relationship, and if the quality verification is passed, determining that the to-be-detected data development code passes the detection. According to the method and the device, the data development codes can be quickly and automatically detected, the detection efficiency of error codes in the data development codes is improved, and the blood relationship between the fields and the indexes can be checked on the business level, so that the business quality verification can be quickly performed on the data development codes.

Description

Technical Field

[0001] The present application belongs to the field of big data technology, and in particular, relates to an automatic detection method, system, device and storage medium for data development codes. Background Art

[0002] With the development of big data technology, data development has become an important part of enterprise data processing. At present, there are some methods for automated code review, and the entire code review process does not require human participation at all. For example, some automated code review tools on the market, such as SonarQube and Checkstyle, are mainly used for code quality analysis of general programming languages. However, existing automated code review tools are usually not specifically targeted at data development scenarios, so they may not meet the needs of specific business scenarios. For example, for specific needs in the data development of the medical big data middle platform, additional manual code quality review based on business scenarios is required, which is not only time-consuming and labor-intensive, but also likely to result in human errors. Summary of the invention

[0003] The purpose of this application is to provide a method, system, device and storage medium for automatic detection of data development codes, which can automatically locate the erroneous part of the data development code, and can view the blood relationship between fields and indicators at the business level, and quickly perform business quality verification on the data development code.

[0004] In a first aspect, the present application provides a method for automatically detecting data development codes, the method comprising:

[0005] Obtain the data to be tested and develop the code;

[0006] Determine whether the data development code to be tested is consistent with the code review rules, if not, modify the data development code to be tested, if consistent, establish a data lineage relationship based on the data development code to be tested;

[0007] The data development code to be tested is quality checked. If the quality check fails, the error code is located and modified based on the data lineage relationship. If the quality check passes, the data development code to be tested passes the test.

[0008] In an implementation of the first aspect, a specification statement of a target type is acquired based on a data development specification; the specification statement of the target type corresponds to the type of the data development code to be detected;

[0009] The conditional variables, logical variables, and grammatical structures in the specification statements of the target type are extracted to generate the code review rules corresponding to the type of the data development code to be detected.

[0010] In an implementation of the first aspect, developing a code based on the data to be detected to establish a data lineage relationship includes:

[0011] Determine the task type corresponding to the data development code to be detected;

[0012] Performing structural analysis on the data development code to be detected based on the task type to obtain multiple SQL statements of different categories;

[0013] A field-level data lineage relationship is established based on the multiple SQL statements of different categories.

[0014] In an implementation of the first aspect, obtaining multiple SQL statements of different categories includes:

[0015] When it is determined that the task type is an entry-to-lake type or an ETL type, the data development code to be detected is parsed using SPARK SQL code to obtain multiple SQL statements;

[0016] Acquire grammatical structure and semantic information of multiple SQL statements;

[0017] The multiple SQL statements are classified based on the grammatical structure and the semantic information to obtain multiple SQL statements of different categories.

[0018] In an implementation of the first aspect, obtaining multiple SQL statements of different categories includes:

[0019] When it is confirmed that the task type is the FineBI type, the SQL code is parsed for the data development code to be detected to obtain multiple SQL statements;

[0020] Acquire the syntax structure and semantic information of the plurality of SQL statements and the used functions in the SQL statements;

[0021] The multiple SQL statements are classified based on the grammatical structure, the semantic information and the usage function to obtain multiple SQL statements of different categories.

[0022] In an implementation of the first aspect, a field-level data lineage relationship is established based on a plurality of SQL statements of different categories:

[0023] Based on the multiple SQL statements of different categories, a first corresponding relationship and a second corresponding relationship are established; the first corresponding relationship is a corresponding relationship between a source table and a target table in the multiple SQL statements, and the second corresponding relationship is a corresponding relationship between a source field and a target field in the multiple SQL statements;

[0024] The field-level data lineage relationship is established based on the first corresponding relationship and the second corresponding relationship.

[0025] In an implementation of the first aspect, performing quality verification on the data development code to be tested includes:

[0026] Determine quality verification rules based on development requirements;

[0027] Obtaining the running result of the data development code to be detected;

[0028] Determine whether the operation result complies with the quality verification rules. If not, locate and modify the error code based on the data lineage relationship. If yes, the quality verification passes.

[0029] In a second aspect, the present application provides an automatic detection system for data development codes, the system comprising:

[0030] An acquisition module is used to acquire the data development code to be detected;

[0031] A code review module is used to determine whether the data development code to be tested is consistent with the code review rules. If not, the data development code to be tested is modified. If consistent, a data lineage relationship is established based on the data development code to be tested.

[0032] The quality verification module is used to perform quality verification on the data development code to be tested. If the quality verification fails, the error code is located and modified based on the data lineage relationship. If the quality verification passes, the data development code to be tested passes the test.

[0033] In a third aspect, the present application provides an electronic device, comprising: a processor and a memory; the memory is used to store a computer program; the processor is used to execute the computer program stored in the memory, so that the electronic device performs the above-mentioned automatic detection method of the data development code.

[0034] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by an electronic device, implements the above-mentioned method for automatically detecting data development codes.

[0035] As described above, the automatic detection method, system, device and storage medium of the data development code described in this application have the following beneficial effects:

[0036] This application adopts an automated code review method. Through the three steps of syntax checking, structure analysis, and quality verification of the code, the different types of code in the task are first unloaded according to the corresponding syntax to complete the basic syntax check, and then the structure of the code is analyzed for specific lake entry tasks, ETL tasks, and FineBI tasks to obtain the field-level blood relationship. Finally, the quality verification rules are configured as required. This can not only ensure that the business data can meet the needs of the business scenario, but also quickly locate the problem link and promptly remind the corresponding personnel to repair it, while also greatly saving manpower and time costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1A Shown is a schematic diagram of an application scenario of the automatic detection method of data development code described in this application.

[0038] Figure 1B Shown are structural diagrams of the client-cloud interaction scenarios in these implementations.

[0039] Figure 2 Shown is a flow chart of an automatic detection method for data development codes according to an embodiment of the present application in one embodiment.

[0040] Figure 3 Shown is a flow chart of an automatic detection method for data development codes according to an embodiment of the present application in one embodiment.

[0041] Figure 4 Shown is a flow chart of an automatic detection method for data development codes according to an embodiment of the present application in one embodiment.

[0042] Figure 5 Shown is a schematic diagram of the principle of an automatic detection method for data development codes described in an embodiment of the present application in one embodiment.

[0043] Figure 6 Shown is a schematic diagram of the structure of an automatic detection system for data development codes described in an embodiment of the present application in one embodiment.

[0044] Figure 7 Shown is a schematic structural diagram of an electronic device of the present application in one embodiment. DETAILED DESCRIPTION

[0045] The following describes the embodiments of the present application through specific examples, and those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. The present application can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0046] It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present application, and thus the drawings only show components related to the present application rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed at will, and the component layout may also be more complicated.

[0047] In addition, in this application, descriptions such as "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in this field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by this application.

[0048] With the development of big data technology, data development has become an important part of enterprise data processing. However, there is a lack of code review tools specifically for data development scenarios. For medical big data platforms, it usually involves a large amount of data flow and processing, covering multiple links such as medical records, diagnosis, treatment plans, and drug management. Since the amount of data on medical big data platforms is very large and often involves multi-level data processing, it is difficult to quickly locate the error links in the code developed specifically for medical big data platforms. This makes it particularly difficult to trace the source of code errors in the complex data flow process of medical big data platforms, making it impossible to guarantee the stability and data quality of medical big data platforms.

[0049] In order to at least solve the above-mentioned problems, an embodiment of the present application provides an automatic detection method for data development code, which can quickly and automatically detect data development code, improve the detection efficiency of error code in data development code, and view the blood relationship between fields and indicators at the business level, so as to quickly perform business quality verification on data development code.

[0050] In some embodiments, the automatic detection method of the data development code provided in the embodiments of the present application can be applied to Figure 1A The data shown is used to develop an automatic detection device for the code. Figure 1A As shown, the automatic detection device 1 for data development code includes a processor 11 and a database 12. The automatic detection method for data development code provided in the embodiment of the present application can be applied to the processor 11.

[0051] in, Figure 1B The processor 11 in the embodiment may be a single processor or a processor cluster or a cloud computing center composed of multiple processors, and the specific details are not limited here. Figure 1B Only one processor 11 and one database 12 are shown, but it should be understood that Figure 1B The examples are only used to understand the present solution, and the specific numbers of processors 11 and databases 12 should be flexibly determined based on actual conditions.

[0052] In some other embodiments, the automatic detection device 1 for data development code may not include the database 12, but only include a processor 11 with a storage function. The automatic detection method for data development code provided in the embodiment of the present application can be applied to the processor 11. The processor 11 with a storage function may include a computer, a mobile phone, etc., which is not limited here.

[0053] In still other embodiments, the automatic detection method of the data development code described in the present application can be applied to the end-cloud interaction scenario. Figure 1B The following is a schematic diagram of the structure of the client-cloud interaction scenario in these implementations. Figure 1B As shown, the terminal-cloud interaction system 2 includes a terminal 20 and a cloud server 21. The terminal 20 and the cloud server 21 can communicate with each other, and the communication method is not limited to wired or wireless.

[0054] Among them, the terminal 20 can be mobile or fixed, for example, the terminal 20 can be a wireless terminal or a wired terminal. The wireless terminal can refer to a device with wireless transceiver function, which can be deployed in various application scenarios. The terminal 20 can be a mobile phone, a laptop computer, etc., which is not limited here. The cloud server 21 can include one or more servers, or one or more processing nodes, or one or more virtual machines running on the server. The cloud server 21 can also be called a server cluster, a management platform, an automatic detection center for data development code, etc., which is not limited in the embodiments of the present application.

[0055] The technical solutions in the embodiments of the present application will be described in detail below in conjunction with the drawings in the embodiments of the present application.

[0056] The following embodiments of the present application provide a method for automatically detecting data development codes, which can be performed by Figure 1A The processor 11 or Figure 1B It is implemented by the cloud server 21 shown. Figure 2 The flowchart of the automatic detection method of the data development code described in the embodiment of the present application is shown as follows: Figure 2 As shown, the automatic detection method of the data development code includes steps S1 to S4.

[0057] S1. Obtain the data to be tested and develop the code.

[0058] In some embodiments, data development tasks may be captured periodically, and data development codes corresponding to the data development tasks may be obtained as the data development codes to be detected.

[0059] S2. Determine whether the data development code to be tested is consistent with the code review rules. If not, modify the data development code to be tested. If consistent, establish a data lineage relationship based on the data development code to be tested.

[0060] Specifically, it is determined whether the conditional variables, logical variables, and grammatical structures in the data development code to be tested are consistent with the corresponding conditional variables and / or logical variables in the code review rules. If they are inconsistent, the code portion that is inconsistent with the code review rules is recorded, and an alarm is sent to the corresponding data developer, who can then modify the inconsistent code portion accordingly according to the code review rules, and continue to determine whether the modified data development code to be tested is consistent with the code review rules, until the data development code to be tested is consistent with the code review rules, and a data lineage relationship is established based on the data development code to be tested.

[0061] In some embodiments, the implementation method of obtaining the code review rules includes: obtaining a specification statement of a target type based on a data development specification; the specification statement of the target type corresponds to the type of the data development code to be tested; extracting conditional variables, logical variables, and grammatical structures in the specification statement of the target type, and generating the code review rules corresponding to the type of the data development code to be tested.

[0062] In some embodiments, the partition table creation rules, field alias naming rules, window function usage rules, data table writing rules, SQL parameter usage rules, where condition rules in SQL, table association rules, UDF function usage rules, and comment rules can be obtained from the development specifications of the data center, and the corresponding types of standard statements can be extracted from the table association method, field conversion relationship, where conditions, etc. in the table model design, and the conditional variables, logical variables, and grammatical structures in the standard statements can be used as the corresponding code review rules. For example, the standard statement can be extracted from the partition table creation rules, and the conditional variables, logical variables, and grammatical structures in the standard statement can be used as the code review rules for the partition table creation rules.

[0063] That is, after the data development task is captured through step S1, the corresponding data development code is obtained, the type of the code is determined, for example, whether the code is the above-mentioned SQL parameter code, etc., and the code type is used as the target type, and the corresponding specification statement is obtained from the data development specification as the specification statement of the target type, and the conditional variables, logical variables, and grammatical structures in different specification statements are used as code review rules of corresponding types.

[0064] Further, when it is determined through step S2 that the data development code to be detected is consistent with the code review rule, a data lineage relationship is established based on the data development code to be detected. Figure 3 The flowchart of the automatic detection method of the data development code described in the embodiment of the present application is shown as follows: Figure 3 As shown, establishing data lineage relationship based on the data to be detected developing code includes steps S21 to S23.

[0065] S21. Determine the task type corresponding to the data development code to be detected.

[0066] Specifically, determine whether the task type of the data development code to be detected is an entry task type, an ETL task type, or a FineBI task type, and use this to perform subsequent structural analysis.

[0067] S22: Perform structural analysis on the data development code to be detected based on the task type to obtain multiple SQL statements of different categories.

[0068] Specifically, when it is determined that the task type is an entry type or an ETL type, the data development code to be detected is parsed by SPARK SQL code to obtain multiple SQL statements. After that, the grammatical structure and semantic information of the multiple SQL statements are obtained, and the multiple SQL statements are classified based on the grammatical structure and the semantic information to obtain multiple SQL statements of different categories.

[0069] In some embodiments, when it is determined that the task type is an entry type or an ETL type, SPARKSQL code parsing is performed. First, the script containing multiple SQL statements in the data development code to be detected is split into multiple separate SQL statements. After that, the separate SQL statements are parsed into corresponding syntax trees to obtain the grammatical structure and semantic information describing the statements. Among them, the grammatical structure contains the grammatical units and grammatical relationship information of the SQL statements, such as keywords, table names, column names, function names, subordinate relationships, connection relationships, etc. The semantic information contains the semantic units and semantic relationship information in the SQL statements, such as tables, columns, constraints, reference relationships, constraint relationships, etc. After the data development code parsing structure to be detected is completed, multiple SQL statements can be divided into 16 different types of data operation types based on the grammatical structure and the semantic information, including DROP_VIEW, CREATE_TABLE, DROP_TABLE, SHOW, SELECT, CREATE_VIEW, INSERT_INTO, SET, DESC, CALL, CREATE_TABLE_AS_SELECT, TRUNCATE_TABLE, ALTER_TABLE, INSERT_OVERWRITE, UPDATE, and DELETE, and each data operation type corresponds to a type of SQL statement.

[0070] In other embodiments, after the code parsing structure for the data to be detected is developed, multiple SQL statements can be subdivided into three types: READ, WRITE, and OTHER according to different data usage scenarios based on the grammatical structure and the semantic information, or multiple SQL statements can be subdivided into three types: OVERWRITE, INTO, and non-write SQL according to different ways of writing data into the table.

[0071] Specifically, when it is confirmed that the task type is the FineBI type, the SQL code is parsed for the data development code to be detected to obtain multiple SQL statements. After that, the grammatical structure, semantic information and usage functions of the multiple SQL statements are obtained, and the multiple SQL statements are classified based on the grammatical structure, the semantic information and the usage functions to obtain multiple SQL statements of different categories.

[0072] In some embodiments, when it is confirmed that the task type is of the FineBI type, the SQL code is parsed for the data development code to be detected. First, the script containing multiple SQL statements in the data development code to be detected is split into multiple separate SQL statements. After that, the separate SQL statements are parsed into corresponding syntax trees to obtain the grammatical structure and semantic information describing the statements. In addition, when it is confirmed that the task type is of the FineBI type, the use functions in the SQL statements are further obtained, such as LEN, SUM, ABS, INDEXOFARRAY, IF, SPLIT, IN, etc. After the parsing structure of the data development code to be detected is completed, the multiple SQL statements can also be classified according to different data operation types, or different data usage situations, or different ways of writing data into the table based on the grammatical structure, the semantic information and the use functions to obtain multiple SQL statements of different categories.

[0073] S23: Establishing field-level data lineage relationships based on the multiple SQL statements of different categories.

[0074] Specifically, based on the multiple SQL statements of different categories, a first correspondence and a second correspondence are established, and based on the first correspondence and the second correspondence, the field-level data lineage relationship is established. The first correspondence is the correspondence between the source table and the target table in the multiple SQL statements, and the second correspondence is the correspondence between the source field and the target field in the multiple SQL statements.

[0075] In some embodiments, after the code parsing structure of the data to be detected is developed, the tables and fields involved in multiple SQL statements are matched according to the source table and the target table to obtain a first corresponding relationship, and the source field and the target field are matched to obtain a second corresponding relationship, and information such as the first corresponding relationship and the second corresponding relationship is stored in a graph database for display, so as to establish field-level data lineage relationships, that is, field-level data lineage relationships can be obtained through query statements.

[0076] S3. Performing quality verification on the data development code to be tested. If the quality verification fails, locating and modifying the error code based on the data lineage relationship. If the quality verification passes, the data development code to be tested passes the test.

[0077] When the data development code to be tested complies with the code review rules, the data development code to be tested must also be quality checked to determine whether the data development code to be tested meets the development requirements or business requirements of a specific scenario. Figure 4The flowchart of the automatic detection method of the data development code described in the embodiment of the present application is shown as follows: Figure 4 As shown, the quality verification of the data development code to be tested includes steps S31 to S33.

[0078] S31. Determine quality verification rules based on development requirements.

[0079] S32: Obtain the running result of the data development code to be detected.

[0080] S33. Determine whether the operation result complies with the quality verification rule. If not, locate and modify the error code based on the data lineage relationship. If yes, the quality verification passes.

[0081] Specifically, according to the development requirements or business requirements, corresponding verification rules are configured for different fields, tables, business scenarios, etc., such as integrity verification, non-null value verification, consistency verification, accuracy verification, timeliness verification, redundancy verification, business rule verification, etc. When the data development code to be tested meets the code review rules, the data development code to be tested is run to obtain the running result, that is, after a series of data calculation tasks are completed, the running result is verified according to the quality verification rules. If the running result meets the quality verification rules, it means that the data development code to be tested can perform the calculation tasks that meet the development requirements or business requirements, and the code quality verification passes. If the running result cannot meet the quality verification rules, it means that a problem occurs in a certain business scenario or the business indicator does not meet the prediction, then the corresponding developer and relevant person in charge can be notified to repair it. Among them, the developer and relevant person in charge can quickly locate and modify the error code according to the data lineage relationship, that is, modify the code part that does not meet the business requirements until the final running result meets the business scenario requirements.

[0082] For example, taking the real-time inventory query function in the business scenario of "Check Inventory Every Day" as an example, when the user selects the provincial company that needs to be viewed, the data development code to be tested is run, and the running result is shown as an option with a null value in the product classification filter. Since it is correct and acceptable for the field value to be null at the code and syntax level, this situation cannot be detected when the data development code to be tested is tested through the code review rule of step S2. However, since the situation that the field value is null when displayed on the front end will reduce the user's experience, it is necessary to further use the quality check of step S3 to detect the null value of the product classification field, that is, to configure the quality check rule of non-null value check. If it is found that there is a null value in the product classification field, it means that it does not meet the quality check rule of non-null value check, and the corresponding quality check result will be sent to the corresponding developer or responsible person for processing.

[0083] In addition, all quality inspection results and records will be saved in the database for later display and statistics.

[0084] Figure 5 The schematic diagram of the principle of the automatic detection method of the data development code described in the embodiment of the present application is shown as follows: Figure 5 As shown, when the data development code is automatically detected, the target type specification statement corresponding to the data development code type will be obtained from the development specification first, and the conditional variables, logical variables, and grammatical structures will be extracted from the target type specification statement to form a code review rule. Afterwards, the data development code to be detected is reviewed based on the code review rules to determine whether the conditional variables, logical variables, and grammatical structures in the data development code to be detected are consistent with the code review rules. If not, the inconsistent code part is recorded and an alarm is issued to prompt the developer to make modifications. If consistent, the data kinship relationship is automatically generated according to the data development code. Afterwards, the quality verification rules are configured based on business needs, and the data development code is run to determine whether the running results meet the quality verification rules. If satisfied, the quality verification results are recorded and saved to the database, and the automatic detection process ends. If not satisfied, the inconsistent code part is recorded and an alarm is issued to prompt the developer to make modifications.

[0085] Therefore, the automatic detection method of data development code provided in the embodiment of the present application, through code review rules and quality verification rules, first unloads different types of codes in the task according to the corresponding syntax, and completes basic syntax checking. When the code review rules are met, the present application then performs structural analysis of the data development code for specific entry tasks, ETL tasks, and FineBI tasks to obtain field-level blood relationships, and finally configures corresponding quality verification rules according to business scenario requirements. The present application can not only ensure that business data can meet the needs of business scenarios, but also quickly locate problem links and promptly remind corresponding personnel to repair them, while also greatly saving manpower costs and time costs.

[0086] The protection scope of the automatic detection method of data development code described in the embodiment of the present application is not limited to the execution order of the steps listed in the present embodiment. All solutions implemented by adding, reducing or replacing steps in the prior art based on the principles of the present application are included in the protection scope of the present application.

[0087] The embodiment of the present application also provides an automatic detection system for data development codes, and the automatic detection system for data development codes can implement the automatic detection method for data development codes described in the present application. However, the implementation device of the automatic detection system for data development codes described in the present application includes but is not limited to the structure of the automatic detection system for data development codes listed in the present embodiment. All structural deformations and replacements of the prior art made according to the principles of the present application are included in the protection scope of the present application.

[0088] Figure 6 The structure diagram of the automatic detection system of the data development code described in the embodiment of the present application is shown as follows: Figure 6 As shown, the automatic detection system 4 for data development code includes an acquisition module 41 , a code review module 42 and a quality verification module 43 .

[0089] The acquisition module 41 is used to acquire the data development code to be detected.

[0090] The code review module 42 is used to determine whether the data development code to be tested is consistent with the code review rules. If not, the data development code to be tested is modified. If consistent, a data lineage relationship is established based on the data development code to be tested.

[0091] The quality verification module 43 is used to perform quality verification on the data development code to be tested. If the quality verification fails, the error code is located and modified based on the data lineage relationship. If the quality verification passes, the data development code to be tested passes the test.

[0092] It should be noted that the structures and principles of the acquisition module 41, the code review module 42 and the quality verification module 43 correspond one-to-one to the steps in the above-mentioned data development code automatic detection method, so they are not described in detail here.

[0093] In the several embodiments provided in the present application, it should be understood that the disclosed system, device or method can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of modules / units is only a logical function division. There may be other division methods in actual implementation. For example, multiple modules or units can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules or units, which can be electrical, mechanical or other forms.

[0094] The modules / units described as separate components may or may not be physically separated, and the components displayed as modules / units may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules / units may be selected according to actual needs to achieve the purpose of the embodiments of the present application. For example, the functional modules / units in the various embodiments of the present application may be integrated into one processing module, or each module / unit may exist physically separately, or two or more modules / units may be integrated into one module / unit.

[0095] Those of ordinary skill in the art should further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0096] The present application embodiment also provides a computer-readable storage medium. A person of ordinary skill in the art can understand that all or part of the steps in the method for implementing the above embodiment can be completed by a program to instruct the processor, and the program can be stored in a computer-readable storage medium, and the storage medium is a non-transitory medium, such as a random access memory, a read-only memory, a flash memory, a hard disk, a solid-state hard disk, a magnetic tape, a floppy disk, an optical disc, and any combination thereof. The above storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or a data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a digital video disc (DVD)), or a semiconductor medium (e.g., a solid-state hard disk (SSD)), etc.

[0097] The embodiment of the present application further provides an electronic device, wherein the electronic device includes a processor and a memory.

[0098] The memory is used to store computer programs.

[0099] The memory includes: ROM, RAM, disk, USB flash drive, memory card or CD and other media that can store program codes.

[0100] The processor is connected to the memory and is used to execute the computer program stored in the memory so that the electronic device executes the above-mentioned automatic detection method of the data development code.

[0101] Preferably, the processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components.

[0102] like Figure 7As shown, the electronic device of the present application is in the form of a general computing device. The components of the electronic device may include but are not limited to: one or more processors or processing units 51, a memory 52, and a bus 53 connecting different system components (including the memory 52 and the processing unit 51).

[0103] Bus 53 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor or a local bus using any of a variety of bus architectures. For example, these architectures include but are not limited to Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MAC) bus, Enhanced ISA bus, Video Electronics Standards Association (VESA) local bus and Peripheral Component Interconnect (PCI) bus.

[0104] Electronic devices typically include a variety of computer system readable media. These media can be any available media that can be accessed by the electronic device, including volatile and non-volatile media, removable and non-removable media.

[0105] The memory 52 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 521 and / or cache memory 522. The electronic device may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 523 may be used to read and write non-removable, non-volatile magnetic media ( Figure 7 not shown, usually called a "hard drive"). Although Figure 7 Not shown, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk"), and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, a DVD-ROM, or other optical medium) may be provided. In these cases, each drive may be connected to the bus 53 via one or more data medium interfaces. The memory 52 may include at least one program product having a set (e.g., at least one) of program modules that are configured to perform the functions of the various embodiments of the present application.

[0106] A program / utility 524 having a set (at least one) of program modules 5241 may be stored, for example, in the memory 52, such program modules 5241 including but not limited to an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment. The program modules 5241 generally perform the functions and / or methods of the embodiments described herein.

[0107] The electronic device may also communicate with one or more external devices (e.g., keyboards, pointing devices, displays, etc.), one or more devices that enable a user to interact with the electronic device, and / or any device that enables the electronic device to communicate with one or more other computing devices (e.g., network cards, modems, etc.). Such communication may be performed via input / output (I / O) interface 54. Furthermore, the electronic device may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via network adapter 55. Figure 7 As shown, the network adapter 55 communicates with other modules of the electronic device via the bus 53. It should be understood that, although not shown in the figure, other hardware and / or software modules can be used in conjunction with the electronic device, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0108] The present application embodiment may also provide a computer program product, the computer program product including one or more computer instructions. When the computer instructions are loaded and executed on a computing device, the process or function described in the embodiment of the present application is generated in whole or in part. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, computer or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer or data center.

[0109] When the computer program product is executed by a computer, the computer executes the method described in the above method embodiment. The computer program product may be a software installation package, and when the above method is required, the computer program product may be downloaded and executed on a computer.

[0110] The descriptions of the processes or structures corresponding to the above-mentioned figures have different emphases. For parts that are not described in detail in a certain process or structure, please refer to the relevant descriptions of other processes or structures.

[0111] The above embodiments are merely illustrative of the principles and effects of the present application and are not intended to limit the present application. Anyone familiar with the technology may modify or change the above embodiments without violating the spirit and scope of the present application. Therefore, all equivalent modifications or changes made by a person of ordinary skill in the art without departing from the spirit and technical ideas disclosed in the present application shall still be covered by the claims of the present application.

Claims

1. A method for automatically detecting data development codes, characterized in that: The method comprises: Obtain the data to be tested and develop the code; Determine whether the data development code to be tested is consistent with the code review rules, if not, modify the data development code to be tested, if consistent, establish a data lineage relationship based on the data development code to be tested; The data development code to be tested is quality checked. If the quality check fails, the error code is located and modified based on the data lineage relationship. If the quality check passes, the data development code to be tested passes the test.

2. The automatic detection method of data development code according to claim 1, characterized in that: The method comprises: Acquire a specification statement of a target type based on a data development specification; the specification statement of the target type corresponds to the type of the data development code to be detected; The conditional variables, logical variables, and grammatical structures in the specification statements of the target type are extracted to generate the code review rules corresponding to the type of the data development code to be detected.

3. The automatic detection method of data development code according to claim 1, characterized in that: Developing codes based on the data to be detected to establish data lineage relationships includes: Determine the task type corresponding to the data development code to be detected; Performing structural analysis on the data development code to be detected based on the task type to obtain multiple SQL statements of different categories; A field-level data lineage relationship is established based on the multiple SQL statements of different categories.

4. The automatic detection method of data development code according to claim 3 is characterized in that: Get multiple SQL statements of different categories including: When it is determined that the task type is an entry-to-lake type or an ETL type, the data development code to be detected is parsed using SPARK SQL code to obtain multiple SQL statements; Obtaining grammatical structure and semantic information of the plurality of SQL statements; The multiple SQL statements are classified based on the grammatical structure and the semantic information to obtain multiple SQL statements of different categories.

5. The automatic detection method of data development code according to claim 3, characterized in that: Get multiple SQL statements of different categories including: When it is confirmed that the task type is the FineBI type, the SQL code is parsed for the data development code to be detected to obtain multiple SQL statements; Acquire the syntax structure and semantic information of the plurality of SQL statements and the used functions in the SQL statements; The multiple SQL statements are classified based on the grammatical structure, the semantic information and the usage function to obtain multiple SQL statements of different categories.

6. The automatic detection method of data development code according to claim 3, characterized in that: A data lineage relationship at the field level is established based on multiple SQL statements of different categories: Based on the multiple SQL statements of different categories, a first corresponding relationship and a second corresponding relationship are established; the first corresponding relationship is a corresponding relationship between a source table and a target table in the multiple SQL statements, and the second corresponding relationship is a corresponding relationship between a source field and a target field in the multiple SQL statements; The field-level data lineage relationship is established based on the first corresponding relationship and the second corresponding relationship.

7. The automatic detection method of data development code according to claim 1, characterized in that: The quality check of the data development code to be tested includes: Determine quality verification rules based on development requirements; Obtaining the running result of the data development code to be detected; Determine whether the operation result complies with the quality verification rules. If not, locate and modify the error code based on the data lineage relationship. If yes, the quality verification passes.

8. An automatic detection system for data development code, characterized in that: The system comprises: An acquisition module is used to acquire the data development code to be detected; A code review module is used to determine whether the data development code to be tested is consistent with the code review rules. If not, the data development code to be tested is modified. If consistent, a data lineage relationship is established based on the data development code to be tested. The quality verification module is used to perform quality verification on the data development code to be tested. If the quality verification fails, the error code is located and modified based on the data lineage relationship. If the quality verification passes, the data development code to be tested passes the test.

9. An automatic detection device for data development code, characterized in that: include: Processor and memory; The memory is used to store computer programs; The processor is used to execute the computer program stored in the memory, so that the electronic device executes the automatic detection method of the data development code according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by an electronic device, the automatic detection method of the data development code described in any one of claims 1 to 7 is implemented.

Images