Method, system and device for automatic detection of data development code and storage medium

Abstract

The application provides an automatic detection method, system and device of data development code and a storage medium. The method comprises the following steps: obtaining data development code to be detected; determining whether the data development code to be detected is consistent with a code review rule; if not, modifying the data development code to be detected; if yes, establishing a data blood relationship based on the data development code to be detected; performing quality checking on the data development code to be detected; if the quality checking fails, locating and modifying error code based on the data blood relationship; and if the quality checking passes, the data development code to be detected passes the detection. The application can quickly and automatically detect data development code, improve the detection efficiency of error code in the data development code, and check the blood relationship between fields and indicators at the business level, so as to quickly perform business quality checking on the data development code.

Description

Technical Field

[0001] This application belongs to the field of big data technology, and in particular relates to an automatic detection method, system, device and storage medium for data development code. Background Technology

[0002] With the development of big data technology, data development has become a crucial component of enterprise data processing. Currently, there are methods for automated code review, eliminating the need for human intervention in the entire process. Examples include automated code review tools like SonarQube and Checkstyle, which are primarily used for code quality analysis in general programming languages. However, existing automated code review tools are often not specifically designed for data development scenarios and may therefore fail to meet the needs of specific business scenarios. For instance, for the specific requirements of developing a medical big data platform, additional human code quality reviews based on the business scenario are still necessary. This is not only time-consuming and labor-intensive but also susceptible to human error. Summary of the Invention

[0003] The purpose of this application is to provide an automatic detection method, system, device, and storage medium for data development code, which can automatically locate the erroneous parts of the data development code and view the lineage relationship between fields and indicators at the business level, and quickly perform business quality verification on the data development code.

[0004] Firstly, this application provides an automatic detection method for data development code, the method comprising:

[0005] Develop code to acquire the data to be tested;

[0006] Determine whether the data development code to be tested is consistent with the code review rules. If it is inconsistent, modify the data development code to be tested. If it is consistent, establish a data lineage based on the data development code to be tested.

[0007] The development code of the data to be tested is subjected to quality verification. If the quality verification fails, the error code is located and modified based on the data lineage. If the quality verification passes, the development code of the data to be tested passes the test.

[0008] In one implementation of the first aspect, a specification statement for the target type is obtained based on the data development specification; the specification statement for the target type corresponds to the type of the data development code to be detected.

[0009] Extract the condition variables, logical variables, and syntax structures from the specification statements of the target type, and generate the code review rules corresponding to the type of the data development code to be inspected.

[0010] In one implementation of the first aspect, establishing data lineage based on the data to be detected through code development includes:

[0011] Determine the task type corresponding to the development code of the data to be detected;

[0012] Based on the task type, the code for developing the data to be detected is structurally parsed to obtain multiple SQL statements of different categories;

[0013] Establish field-level data lineage relationships based on multiple SQL statements of different categories.

[0014] In one implementation of the first aspect, retrieving multiple SQL statements of different categories includes:

[0015] When the task type is determined to be either lake entry type or ETL type, SPARK SQL code parsing is performed on the development code of the data to be detected to obtain multiple SQL statements;

[0016] Obtain the syntax structure and semantic information of multiple SQL statements;

[0017] Based on the grammatical structure and semantic information, multiple SQL statements are classified to obtain multiple SQL statements of different categories.

[0018] In one implementation of the first aspect, retrieving multiple SQL statements of different categories includes:

[0019] When the task type is confirmed to be FineBI, the SQL code development code of the data to be tested is parsed to obtain multiple SQL statements;

[0020] Obtain the syntax structure, semantic information, and functions used in the SQL statements;

[0021] Based on the grammatical structure, the semantic information, and the usage function, multiple SQL statements are classified to obtain multiple SQL statements of different categories.

[0022] In one implementation of the first aspect, field-level data lineage is established based on multiple SQL statements of different categories:

[0023] Based on multiple SQL statements of different categories, a first correspondence and a second correspondence are established; the first correspondence is the correspondence between the source table and the target table in the multiple SQL statements, and the second correspondence is the correspondence between the source field and the target field in the multiple SQL statements.

[0024] The field-level data lineage relationship is established based on the first correspondence and the second correspondence.

[0025] In one implementation of the first aspect, quality verification of the development code for the data to be tested includes:

[0026] Quality verification rules are determined based on development requirements;

[0027] Obtain the execution result of the code developed for the data to be tested;

[0028] Determine whether the running result conforms to the quality verification rules. If it does not conform, locate and modify the error code based on the data lineage. If it conforms, the quality verification passes.

[0029] Secondly, this application provides an automatic detection system for data development code, the system comprising:

[0030] The acquisition module is used to acquire the data to be detected.

[0031] The code review module is used to determine whether the data development code to be tested is consistent with the code review rules. If it is inconsistent, the data development code to be tested is modified. If it is consistent, a data lineage relationship is established based on the data development code to be tested.

[0032] The quality verification module is used to perform quality verification on the development code of the data to be tested. If the quality verification fails, the error code is located and modified based on the data lineage. If the quality verification passes, the development code of the data to be tested passes the test.

[0033] Thirdly, this application provides an electronic device, the electronic device comprising: a processor and a memory; the memory for storing a computer program; the processor for executing the computer program stored in the memory, so that the electronic device executes the above-described automatic detection method for data development code.

[0034] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by an electronic device, implements the above-described automatic detection method for data development code.

[0035] As described above, the automatic detection method, system, device, and storage medium for data development code described in this application have the following beneficial effects:

[0036] This application employs an automated code review method, which involves three main steps: syntax checking, structural analysis, and quality verification. First, different types of code in the task are sorted according to their corresponding syntax to complete basic syntax checks. Then, the code structure of specific inbound tasks, ETL tasks, and FineBI tasks is analyzed to obtain field-level lineage relationships. Finally, quality verification rules are configured according to requirements. This method not only ensures that business data meets the needs of business scenarios but also quickly locates problematic areas and promptly alerts the relevant personnel for repair, while also significantly reducing labor and time costs. Attached Figure Description

[0037] Figure 1A The diagram shows an application scenario of the automatic detection method for data development code described in this application.

[0038] Figure 1B This diagram illustrates the structure of the mid-cloud interaction scenario in these implementation methods.

[0039] Figure 2 The diagram shown is a flowchart of an embodiment of the automatic detection method for data development code described in this application.

[0040] Figure 3 The diagram shown is a flowchart of an embodiment of the automatic detection method for data development code described in this application.

[0041] Figure 4 The diagram shown is a flowchart of an embodiment of the automatic detection method for data development code described in this application.

[0042] Figure 5 The diagram shown is a schematic representation of one embodiment of the automatic detection method for data development code described in this application.

[0043] Figure 6 The diagram shown is a structural schematic of an embodiment of the automatic detection system for data development code described in this application.

[0044] Figure 7 The diagram shown is a structural schematic of the electronic device of this application in one embodiment. Detailed Implementation

[0045] The following specific examples illustrate the implementation of this application. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. This application can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this application. It should be noted that, unless otherwise specified, the following embodiments and features in the embodiments can be combined with each other.

[0046] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of this application. Therefore, the drawings only show the components related to this application and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0047] Furthermore, the use of terms such as "first" and "second" in this application is for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of those features. Additionally, the technical solutions of the various embodiments can be combined with each other, but only on the basis of being achievable by those skilled in the art. If the combination of technical solutions is contradictory or impossible to implement, such a combination of technical solutions should be considered non-existent and not within the scope of protection claimed in this application.

[0048] With the development of big data technology, data development has become an important part of enterprise data processing. However, there is currently a lack of code review tools specifically designed for data development scenarios. For medical big data platforms, this typically involves massive data flow and processing, covering multiple stages such as medical records, diagnoses, treatment plans, and medication management. Due to the sheer volume of data on medical big data platforms and the often multi-layered data processing involved, it is difficult to quickly locate errors in the code specifically developed for medical big data platforms. This makes tracing the source of code errors particularly difficult during the complex data flow of medical big data platforms, thus compromising the stability and data quality of the platform.

[0049] To at least address the aforementioned issues, embodiments of this application provide an automatic detection method for data development code. This method can quickly and automatically detect data development code, improve the efficiency of detecting errors in data development code, and allow for viewing the lineage relationships between fields and metrics at the business level, thereby enabling rapid business quality verification of the data development code.

[0050] In some embodiments, the automatic detection method for data development code provided in this application can be applied to, for example... Figure 1A The data development code automatic detection device shown. For example... Figure 1A As shown, the automatic detection device 1 for data development code includes a processor 11 and a database 12. The automatic detection method for data development code provided in this application embodiment can be applied to the processor 11.

[0051] in, Figure 1B The processor 11 in the text can be a single processor, a processor cluster consisting of multiple processors, or a cloud computing center, etc., and is not specifically limited here. Although Figure 1B Only one processor 11 and one database 12 are shown, but it should be understood that... Figure 1B The examples in this paper are only for understanding this solution. The specific number of processors 11 and databases 12 should be flexibly determined based on the actual situation.

[0052] In some other embodiments, the automatic detection device 1 for data development code may not include the database 12, but only a processor 11 with storage function. The automatic detection method for data development code provided in this application embodiment can be applied to the processor 11. The processor 11 with storage function may include a computer, mobile phone, etc., and is not limited here.

[0053] In some other embodiments, the automatic detection method for data development code described in this application can be applied to edge-cloud interaction scenarios. Figure 1B This diagram illustrates the structure of the endpoint-cloud interaction scenario in these implementation methods. For example... Figure 1B As shown, the terminal-cloud interaction system 2 includes a terminal 20 and a cloud server 21. The terminal 20 and the cloud server 21 can communicate with each other, and the communication method is not limited to wired or wireless.

[0054] The terminal 20 can be mobile or fixed. For example, it can be a wireless terminal or a wired terminal. A wireless terminal can refer to a device with wireless transceiver capabilities and can be deployed in various application scenarios. The terminal 20 can be a mobile phone, a laptop computer, etc., and is not limited thereto. The cloud server 21 can include one or more servers, or one or more processing nodes, or one or more virtual machines running on the server. The cloud server 21 can also be referred to as a server cluster, a management platform, an automatic detection center for data development code, etc., and is not limited thereto in this embodiment.

[0055] The technical solutions in the embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0056] The following embodiments of this application provide an automatic detection method for data development code, which, for example, can be achieved through... Figure 1A The processor 11 shown or Figure 1B The cloud server 21 shown is used to implement this. Figure 2 The diagram shown is a flowchart illustrating the automatic detection method for data development code described in an embodiment of this application. Figure 2 As shown, the automatic detection method for the data development code includes steps S1 to S4.

[0057] S1. Develop code to obtain the data to be tested.

[0058] In some embodiments, data development tasks can be periodically captured, and the data development code corresponding to the data development tasks can be obtained as the data development code to be tested.

[0059] S2. Determine whether the data development code to be tested is consistent with the code review rules. If it is inconsistent, modify the data development code to be tested. If it is consistent, establish a data lineage based on the data development code to be tested.

[0060] Specifically, it is determined whether the condition variables, logical variables, and syntax structures in the data development code to be tested are consistent with the corresponding condition variables and / or logical variables in the code review rules. If they are inconsistent, the inconsistent code portion is recorded and an alert is sent to the corresponding data developer. The data developer can then modify the inconsistent code portion according to the code review rules, and the modified data development code to be tested is further evaluated to ensure consistency with the code review rules. This process continues until the data development code to be tested is consistent with the code review rules, at which point a data lineage relationship is established based on the data development code to be tested.

[0061] In some embodiments, obtaining the code review rules includes: obtaining specification statements for a target type based on data development specifications; the specification statements for the target type correspond to the type of the data development code to be inspected; extracting condition variables, logical variables, and syntax structures from the specification statements for the target type to generate the code review rules corresponding to the type of the data development code to be inspected.

[0062] In some embodiments, the development specifications of the data platform can be used to extract corresponding standard statements, such as partition table creation rules, field alias naming rules, window function usage rules, data table writing rules, SQL parameter usage rules, SQL WHERE condition rules, table association rules, UDF function usage rules, and comment rules. Additionally, the table model design can include table association methods, field conversion relationships, and WHERE conditions. The condition variables, logical variables, and syntax structures in these standard statements can then be used as corresponding code review rules. For example, standard statements can be extracted from partition table creation rules, and the condition variables, logical variables, and syntax structures within these standard statements can be used as code review rules for this type of code.

[0063] That is, after capturing the data development task through step S1, the corresponding data development code is obtained, the type of the code is determined, such as whether the code is the SQL parameter code mentioned above, and the code type is taken as the target type. The corresponding specification statement is obtained from the data development specification and taken as the specification statement of the target type. The condition variables, logical variables and syntax structures in different specification statements are taken as the code review rules of the corresponding type.

[0064] Furthermore, when step S2 determines that the data development code to be detected is consistent with the code review rules, a data lineage relationship is established based on the data development code to be detected. Figure 3 The diagram shown is a flowchart illustrating the automatic detection method for data development code described in an embodiment of this application. Figure 3 As shown, establishing data lineage based on the data to be detected involves steps S21 to S23.

[0065] S21. Determine the task type corresponding to the data development code to be detected.

[0066] Specifically, the task type of the data development code to be detected is determined to be either a lake entry task type, an ETL task type, or a FineBI task type, and the subsequent structure parsing is performed accordingly.

[0067] S22. Based on the task type, perform structural parsing on the data development code to be detected to obtain multiple SQL statements of different categories.

[0068] Specifically, when the task type is determined to be either a lake entry type or an ETL type, the development code of the data to be tested is parsed using SPARK SQL code parsing to obtain multiple SQL statements. Then, the syntax structure and semantic information of the multiple SQL statements are obtained, and based on the syntax structure and semantic information, the multiple SQL statements are classified to obtain multiple SQL statements of different categories.

[0069] In some embodiments, when the task type is determined to be either an inflow type or an ETL type, SPARKSQL code parsing is performed. First, the script containing multiple SQL statements in the data development code to be tested is split into multiple individual SQL statements. Then, each individual SQL statement is parsed into a corresponding syntax tree to obtain the syntactic structure and semantic information describing the statement. The syntactic structure includes the syntactic units and syntactic relationships of the SQL statement, such as keywords, table names, column names, function names, dependency relationships, and join relationships. The semantic information includes the semantic units and semantic relationships within the SQL statement, such as tables, columns, constraints, reference relationships, and constraint relationships. Once the code parsing structure for the data to be tested is completed, multiple SQL statements can be divided into 16 different data operation types based on the syntax structure and semantic information, including DROP_VIEW, CREATE_TABLE, SHOW, SELECT, CREATE_VIEW, INSERT_INTO, SET, DESC, CALL, CREATE_TABLE_AS_SELECT, TRUNCATE_TABLE, ALTER_TABLE, INSERT_OVERWRITE, UPDATE, and DELETE. Each data operation type corresponds to one type of SQL statement.

[0070] In other embodiments, after the code parsing structure of the data to be detected is completed, multiple SQL statements can be subdivided into three types according to different data usage scenarios: READ, WRITE, and OTHER, based on the syntax structure and semantic information. Alternatively, multiple SQL statements can be subdivided into three types according to different ways of writing data into the table: OVERWRITE, INTO, and non-write SQL.

[0071] Specifically, when the task type is confirmed to be FineBI, the SQL code of the data development code to be tested is parsed to obtain multiple SQL statements. Then, the syntax structure, semantic information, and functions used in the multiple SQL statements are obtained, and the multiple SQL statements are classified based on the syntax structure, semantic information, and functions used to obtain multiple SQL statements of different categories.

[0072] In some embodiments, when the task type is confirmed to be FineBI, the data development code to be tested is parsed using SQL code. First, the script containing multiple SQL statements in the data development code to be tested is split into multiple individual SQL statements. Then, the individual SQL statements are parsed into corresponding syntax trees to obtain the syntactic structure and semantic information describing the statements. In addition, when the task type is confirmed to be FineBI, the functions used in the SQL statements are further obtained, such as LEN, SUM, ABS, INDEXOFARRAY, IF, SPLIT, IN, etc. After the structure of the data development code to be tested is parsed, the multiple SQL statements can also be classified according to different data operation types, different data usage scenarios, or different ways of writing data into the table, based on the syntactic structure, the semantic information, and the functions used, to obtain multiple SQL statements of different categories.

[0073] S23. Establish field-level data lineage relationships based on multiple SQL statements of different categories.

[0074] Specifically, based on multiple SQL statements of different categories, a first correspondence and a second correspondence are established, and the field-level data lineage is established based on the first correspondence and the second correspondence. The first correspondence is the correspondence between the source table and the target table in the multiple SQL statements, and the second correspondence is the correspondence between the source field and the target field in the multiple SQL statements.

[0075] In some embodiments, after the structure of the data to be detected is parsed by the code, the tables and fields involved in multiple SQL statements are matched according to the source table and the target table to obtain a first correspondence relationship, and according to the source field and the target field to obtain a second correspondence relationship. The information such as the first correspondence relationship and the second correspondence relationship is stored in the graph database for display, thereby establishing a field-level data lineage relationship. That is, the field-level data lineage relationship can be obtained through query statements.

[0076] S3. Perform quality verification on the data development code to be tested. If the quality verification fails, locate and modify the error code based on the data lineage. If the quality verification passes, the data development code to be tested passes the test.

[0077] Once the data development code to be tested complies with the code review rules, a quality verification of the data development code to be tested is also required to determine whether the data development code to be tested meets the development requirements or business requirements of a specific scenario. Figure 4The diagram shown is a flowchart illustrating the automatic detection method for data development code described in an embodiment of this application. Figure 4 As shown, the quality verification of the data development code to be tested includes steps S31 to S33.

[0078] S31. Determine quality verification rules based on development requirements.

[0079] S32. Obtain the running result of the development code for the data to be detected.

[0080] S33. Determine whether the running result conforms to the quality verification rules. If it does not conform, locate and modify the error code based on the data lineage. If it conforms, the quality verification passes.

[0081] Specifically, corresponding validation rules are configured for different fields, tables, and business scenarios according to development or business requirements, such as integrity validation, non-null value validation, consistency validation, accuracy validation, timeliness validation, redundancy validation, and business rule validation. Once the data development code to be tested meets the code review rules, it is run to obtain the execution results. That is, after a series of data calculation tasks are completed, the execution results are validated according to the quality validation rules. If the execution results meet the quality validation rules, it means that the data development code to be tested can execute calculation tasks that meet development or business requirements, and the code quality validation passes. If the execution results do not meet the quality validation rules, it means that a problem has occurred in a certain business scenario or that business indicators do not meet predictions, and the corresponding developers and relevant personnel can be notified for repair. The developers and relevant personnel can quickly locate and modify the erroneous code based on the data lineage, that is, modify the code parts that do not meet business requirements until the final execution results meet the business scenario requirements.

[0082] For example, taking the real-time inventory query function in the "Daily Inventory View" business scenario as an example, when a user selects the provincial company they want to view, the code to be tested runs. The result is that empty values ​​appear in the product category filter. Since empty field values ​​are correct and acceptable at the code and syntax level, this situation cannot be detected when the code review rules in step S2 are used to check the code to be tested. However, because empty field values ​​degrade the user experience when displayed on the front end, further quality checks in step S3 are needed to perform empty value detection on the product category field, i.e., configuring non-empty value verification quality checks. If empty values ​​are found in the product category field, it means that the non-empty value verification quality check rule is not met, and the corresponding quality check result will be sent to the relevant developer or responsible person for processing.

[0083] In addition, all quality verification results and records will be saved to the database for future display and statistical analysis.

[0084] Figure 5 The diagram shown is a schematic representation of the principle of the automatic detection method for data development code described in the embodiments of this application. Figure 5 As shown, when automatic detection of data development code begins, the system first retrieves the target type specification statement corresponding to the data development code type from the development specification, and extracts condition variables, logical variables, and syntax structures from the target type specification statement to form code review rules. Then, the data development code to be inspected is reviewed based on these code review rules to determine whether the condition variables, logical variables, and syntax structures in the code are consistent with the code review rules. If they are inconsistent, the inconsistent code portions are recorded and an alert is issued, prompting the developer to make modifications. If they are consistent, data lineage is automatically generated based on the data development code. Next, quality verification rules are configured based on business requirements, and the data development code is run to determine whether the running results meet the quality verification rules. If they do, the quality verification results are recorded and saved to the database, ending the automatic detection process. If they do not meet the rules, the inconsistent code portions are recorded and an alert is issued, prompting the developer to make modifications.

[0085] Therefore, the automatic detection method for data development code provided in this application first performs basic syntax checks by sorting and unscrambling different types of code in the task according to their corresponding syntax, based on code review rules and quality verification rules. If the code review rules are met, this application then performs structural analysis of the data development code for specific data entry tasks, ETL tasks, and FineBI tasks to obtain field-level lineage relationships. Finally, it configures corresponding quality verification rules according to business scenario requirements. This application not only ensures that business data meets the needs of the business scenario but also quickly locates problematic areas and promptly alerts relevant personnel for repair, while significantly reducing labor and time costs.

[0086] The scope of protection of the automatic detection method for data development code described in this application is not limited to the execution order of the steps listed in this embodiment. Any solution implemented by adding, subtracting, or replacing steps in the prior art based on the principles of this application is included within the scope of protection of this application.

[0087] This application also provides an automatic detection system for data development code. The automatic detection system for data development code can implement the automatic detection method for data development code described in this application. However, the implementation device of the automatic detection system for data development code described in this application includes, but is not limited to, the structure of the automatic detection system for data development code listed in this embodiment. All structural modifications and substitutions of the prior art made based on the principles of this application are included within the protection scope of this application.

[0088] Figure 6 The diagram shown is a structural schematic of the automatic detection system for data development code described in an embodiment of this application. Figure 6 As shown, the automatic detection system 4 for data development code includes an acquisition module 41, a code review module 42, and a quality verification module 43.

[0089] Module 41 is used to acquire the data to be detected.

[0090] The code review module 42 is used to determine whether the data development code to be tested is consistent with the code review rules. If it is inconsistent, the data development code to be tested is modified. If it is consistent, a data lineage relationship is established based on the data development code to be tested.

[0091] The quality verification module 43 is used to perform quality verification on the data development code to be tested. If the quality verification fails, the error code is located and modified based on the data lineage. If the quality verification passes, the data development code to be tested passes the test.

[0092] It should be noted that the structure and principle of the acquisition module 41, the code review module 42 and the quality verification module 43 correspond one-to-one with the steps in the above-mentioned automatic detection method for data development code, so they will not be described in detail here.

[0093] In the embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, or methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of modules / units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or units may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection of apparatuses or modules or units may be electrical, mechanical, or other forms.

[0094] The modules / units described as separate components may or may not be physically separate. The components shown as modules / units may or may not be physical modules; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules / units can be selected to achieve the objectives of the embodiments of this application, depending on actual needs. For example, the functional modules / units in the various embodiments of this application may be integrated into one processing module, or each module / unit may exist physically separately, or two or more modules / units may be integrated into one module / unit.

[0095] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0096] This application also provides a computer-readable storage medium. Those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing a processor. The program can be stored in a computer-readable storage medium, which is a non-transitory medium, such as random access memory, read-only memory, flash memory, hard disk, solid-state drive, magnetic tape, floppy disk, optical disk, and any combination thereof. The storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., digital video disc (DVD)), or a semiconductor medium (e.g., solid-state drive (SSD)).

[0097] This application also provides an electronic device. The electronic device includes a processor and a memory.

[0098] The memory is used to store computer programs.

[0099] The memory includes various media capable of storing program code, such as ROM, RAM, magnetic disk, USB flash drive, memory card, or optical disk.

[0100] The processor is connected to the memory and is used to execute the computer program stored in the memory, so that the electronic device executes the above-described automatic detection method for data development code.

[0101] Preferably, the processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0102] like Figure 7As shown, the electronic device of this application is embodied in the form of a general-purpose computing device. The components of the electronic device may include, but are not limited to: one or more processors or processing units 51, memory 52, and bus 53 connecting different system components (including memory 52 and processing unit 51).

[0103] Bus 53 represents one or more of several bus architectures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus using any of the various bus architectures. Examples of these architectures include, but are not limited to, the Industry Standard Architecture (ISA) bus, the Micro Channel Architecture (MAC) bus, the Enhanced ISA bus, the Video Electronics Standards Association (VESA) local bus, and the Peripheral Component Interconnect (PCI) bus.

[0104] Electronic devices typically include a variety of computer-readable media. These media can be any available media that can be accessed by the electronic device, including volatile and non-volatile media, and removable and non-removable media.

[0105] Memory 52 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 521 and / or cache memory 522. The electronic device may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 523 may be used to read and write non-removable, non-volatile magnetic media (…). Figure 7 Not shown; usually referred to as a "hard drive"). Although Figure 7 Not shown, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk") and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to bus 53 via one or more data media interfaces. Memory 52 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of the embodiments of this application.

[0106] A program / utility 524 having a set (at least one) of program modules 5241 may be stored, for example, in memory 52. ​​Such program modules 5241 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment. Program modules 5241 typically perform the functions and / or methods described in the embodiments of this application.

[0107] The electronic device can also communicate with one or more external devices (e.g., keyboard, pointing device, display, etc.), one or more devices that enable a user to interact with the electronic device, and / or any device that enables the electronic device to communicate with one or more other computing devices (e.g., network interface card, modem, etc.). This communication can be performed through input / output (I / O) interface 54. Furthermore, the electronic device can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) through network adapter 55. Figure 7 As shown, network adapter 55 communicates with other modules of the electronic device via bus 53. It should be understood that, although not shown in the figure, other hardware and / or software modules can be used in conjunction with the electronic device, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0108] This application embodiment may also provide a computer program product comprising one or more computer instructions. When the computer instructions are loaded and executed on a computing device, all or part of the processes or functions described in this application embodiment are generated. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means.

[0109] When the computer program product is executed by a computer, the computer performs the method described in the foregoing method embodiments. The computer program product can be a software installation package; when the foregoing method is required, the computer program product can be downloaded and executed on the computer.

[0110] The descriptions of the processes or structures corresponding to the above figures each have their own emphasis. For parts of a process or structure that are not described in detail, please refer to the relevant descriptions of other processes or structures.

[0111] The above embodiments are merely illustrative of the principles and effects of this application and are not intended to limit this application. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of this application. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in this application should still be covered by the claims of this application.

Claims

1. An automatic detection method for data development code, characterized in that, The method includes: Develop code to acquire the data to be tested; Determine whether the data development code to be tested is consistent with the code review rules. If it is inconsistent, modify the data development code to be tested. If it is consistent, establish a data lineage based on the data development code to be tested. The development code for the data to be tested undergoes quality verification. If the quality verification fails, the erroneous code is located and modified based on the data lineage. If the quality verification passes, the development code for the data to be tested passes the test. Establishing the data lineage based on the development code for the data to be tested includes: Determine the task type corresponding to the development code of the data to be detected; Based on the task type, the code for developing the data to be detected is structurally parsed to obtain multiple SQL statements of different categories; Establishing field-level data lineage relationships based on multiple SQL statements of different categories includes: establishing a first correspondence and a second correspondence based on multiple SQL statements of different categories; the first correspondence is the correspondence between the source table and the target table in the multiple SQL statements, and the second correspondence is the correspondence between the source field and the target field in the multiple SQL statements; establishing the field-level data lineage relationships based on the first correspondence and the second correspondence; wherein, obtaining multiple SQL statements of different categories includes: When the task type is determined to be either lake entry type or ETL type, SPARK SQL code parsing is performed on the data development code to be detected to obtain multiple SQL statements; the syntax structure and semantic information of the multiple SQL statements are obtained; the multiple SQL statements are classified based on the syntax structure and semantic information to obtain multiple SQL statements of different categories; When the task type is confirmed to be FineBI type, the SQL code of the data development code to be tested is parsed to obtain multiple SQL statements; the syntax structure, semantic information and functions used in the multiple SQL statements are obtained; the multiple SQL statements are classified based on the syntax structure, semantic information and functions used to obtain multiple SQL statements of different categories.

2. The automatic detection method for data development code according to claim 1, characterized in that, The method includes: The specification statement for the target type is obtained based on the data development specification; the specification statement for the target type corresponds to the type of the data development code to be detected. Extract the condition variables, logical variables, and syntax structures from the specification statements of the target type, and generate the code review rules corresponding to the type of the data development code to be inspected.

3. The automatic detection method for data development code according to claim 1, characterized in that, The quality verification of the development code for the data to be tested includes: Quality verification rules are determined based on development requirements; Obtain the execution result of the code developed for the data to be tested; Determine whether the running result conforms to the quality verification rules. If it does not conform, locate and modify the error code based on the data lineage. If it conforms, the quality verification passes.

4. An automatic detection system for data development code, characterized in that, The system includes: The acquisition module is used to acquire the data to be detected. The code review module is used to determine whether the data development code to be tested is consistent with the code review rules. If it is inconsistent, the data development code to be tested is modified. If it is consistent, a data lineage relationship is established based on the data development code to be tested. The quality verification module is used to perform quality verification on the data development code to be tested. If the quality verification fails, the error code is located and modified based on the data lineage. If the quality verification passes, the data development code to be tested passes the test. Establishing the data lineage based on the data development code to be tested includes: Determine the task type corresponding to the development code of the data to be detected; Based on the task type, the code for developing the data to be detected is structurally parsed to obtain multiple SQL statements of different categories; Establishing field-level data lineage relationships based on multiple SQL statements of different categories includes: establishing a first correspondence and a second correspondence based on multiple SQL statements of different categories; the first correspondence is the correspondence between the source table and the target table in the multiple SQL statements, and the second correspondence is the correspondence between the source field and the target field in the multiple SQL statements; establishing the field-level data lineage relationships based on the first correspondence and the second correspondence; wherein, obtaining multiple SQL statements of different categories includes: When the task type is determined to be either lake entry type or ETL type, SPARK SQL code parsing is performed on the data development code to be detected to obtain multiple SQL statements; the syntax structure and semantic information of the multiple SQL statements are obtained; the multiple SQL statements are classified based on the syntax structure and semantic information to obtain multiple SQL statements of different categories; When the task type is confirmed to be FineBI type, the SQL code of the data development code to be tested is parsed to obtain multiple SQL statements; the syntax structure, semantic information and functions used in the multiple SQL statements are obtained; the multiple SQL statements are classified based on the syntax structure, semantic information and functions used to obtain multiple SQL statements of different categories.

5. An electronic device, characterized in that, include: Processor and memory; The memory is used to store computer programs; The processor is used to execute the computer program stored in the memory to cause the electronic device to perform the automatic detection method of data development code according to any one of claims 1-3.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by an electronic device, the program implements the automatic detection method for data development code as described in any one of claims 1-3.

Images