Code verification method, system and device for data governance and storage medium

Abstract

The invention discloses a data governance-oriented code verification method, system and device and a storage medium, and relates to the technical field of computers, the data governance-oriented code verification method comprises the steps that a verification rule is configured in a database configuration table in advance according to a service demand and / or the verification rule is formulated through a custom code, and the verification rule can be flexibly customized, added and modified according to the service demand; after the code file is submitted to the verification system, performing multi-thread verification on the code file according to each pre-configured verification rule, determining problem codes which do not conform to the verification rules in the code file, and improving the code verification speed; the professional knowledge base is called to provide code repair suggestions for the problem codes, and a more visual verification report including the problem codes and the code repair suggestions is generated, so that a user can understand the report conveniently, the problem codes are quickly positioned to ensure the code quality, the problem processing difficulty is reduced, and the repair efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a code verification method, system, device and storage medium for data governance. Background Art

[0002] In the context of digital transformation, the scale of enterprise data continues to grow, and data governance has become a core component in ensuring data quality and security. The core goal of data governance is to ensure data quality and consistency. As a core tool for database operations, the quality of SQL (Structured Query Language) code directly impacts database performance and security. Low-quality, non-standard, and erroneous SQL code can lead to data inconsistencies, performance degradation, and even system crashes. Therefore, SQL code verification is essential. Summary of the Invention

[0003] The purpose of the present invention is to provide a code verification method, system, device and storage medium for data governance, which can flexibly customize, add and modify verification rules according to business needs, use verification rules to verify code files and generate intuitive verification reports, so that users can deal with problem codes in a timely manner to ensure the correctness of the code.

[0004] To solve the above technical problems, the present invention provides a code verification method for data governance, which includes:

[0005] When a code file is detected to be submitted to the verification system, the API interface is called to obtain the file parameter information of the code file, and the code file is downloaded to a specified location according to the file parameter information;

[0006] Performing multi-threaded verification on the code file according to pre-configured verification rules to determine problematic codes in the code file that do not comply with the verification rules, wherein the verification rules include rules configured by the user in a database configuration table according to business requirements and / or verification rules formulated through custom code, and the database configuration table is a configuration table pre-created in the database for storing the verification rules;

[0007] The professional knowledge base is called upon to provide code repair suggestions for the problem code, and a verification report including the problem code and the code repair suggestions is generated.

[0008] Optionally, after downloading the code file to a designated location according to the file parameter information, the method further includes:

[0009] Determining the file type of the code file, and judging whether the file type of the code file is within a preset verification range;

[0010] When it is determined that the code file is within the preset verification range, the step of performing multi-threaded verification on the code file according to pre-configured verification rules is entered.

[0011] Optionally, after downloading the code file to a designated location according to the file parameter information, the method further includes:

[0012] The code file is preprocessed, and after the preprocessing is completed, the step of performing multi-threaded verification on the code file according to pre-configured verification rules is entered, wherein the preprocessing includes code file format standardization and / or code file syntax parsing.

[0013] Optionally, submitting the detection code file to a verification system includes:

[0014] Periodically poll the verification system at a preset time interval to detect whether there is a new code file submitted to the verification system, or determine whether there is a new code file submitted to the verification system by monitoring whether a code submission event is received, or determine whether there is a new code file submitted to the verification system after detecting that a specified API interface is called by a user, and enter the step of calling the API interface to obtain file parameter information of the code file, wherein the code submission event is an event triggered when the user submits a code file.

[0015] Optionally, the verification rules include any one or more of syntax and structure verification rules, naming standardization verification rules, security and compliance verification rules, performance optimization rules, maintainability and readability verification rules, and industry-specific rules.

[0016] Optionally, after generating the verification report including the problem code and the code repair suggestion, the method further includes:

[0017] The verification reports corresponding to each code file in the same batch are summarized and compressed, and sent to the client to prompt the user to deal with the problem code in time.

[0018] Optionally, after generating the verification report including the problem code and the code repair suggestion, the method further includes:

[0019] The verification report corresponding to each code file is stored, and the verification reports corresponding to each code file in the same batch are statistically analyzed to determine the frequency of each verification rule that verifies the problem code, and generate a visual statistical report.

[0020] To solve the above technical problems, the present invention also provides a code verification system for data governance, including:

[0021] A file download module is used to call an API interface to obtain file parameter information of the code file when detecting that the code file is submitted to the verification system, and download the code file to a specified location according to the file parameter information;

[0022] a verification module, configured to perform multi-threaded verification on the code file according to pre-configured verification rules, and determine problematic code in the code file that does not comply with the verification rules, wherein the verification rules include rules configured by the user in a database configuration table according to business requirements and / or verification rules formulated through custom code, and the database configuration table is a configuration table pre-created in the database for storing the verification rules;

[0023] The report generation module is used to call the professional knowledge base to provide code repair suggestions for the problem code and generate a verification report including the problem code and the code repair suggestions.

[0024] To solve the above technical problems, the present invention also provides a code verification device for data governance, comprising:

[0025] Memory for storing computer programs;

[0026] A processor is configured to implement the steps of any of the above-mentioned code verification methods for data governance when executing the computer program.

[0027] In order to solve the above technical problems, the present invention also provides a storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned code verification methods for data governance are implemented.

[0028] The beneficial effects of the present invention are that it provides a code verification method, system, device and storage medium for data governance, including pre-configuring verification rules in the database configuration table according to business needs and / or formulating verification rules through custom codes, and being able to flexibly customize, add and modify verification rules according to business needs; after the code file is submitted to the verification system, multi-threaded verification is performed on the code file according to the pre-configured verification rules to determine the problem code in the code file that does not comply with the verification rules, thereby improving the code verification speed; calling the professional knowledge base to provide code repair suggestions for the problem code, and generating a more intuitive verification report including the problem code and code repair suggestions, which is convenient for users to understand the report, quickly locate the problem code to ensure code quality, reduce the difficulty of handling problems, and improve repair efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the prior art and the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0030] Figure 1 A first flow chart of a code verification method for data governance provided by the present invention;

[0031] Figure 2 A second flow chart of a code verification method for data governance provided by the present invention;

[0032] Figure 3 A schematic diagram of the structure of a code verification system for data governance provided by the present invention;

[0033] Figure 4 A schematic diagram of the structure of a code verification device for data governance provided by the present invention. DETAILED DESCRIPTION

[0034] The core of the present invention is to provide a code verification method, system, device and storage medium for data governance, which can flexibly customize, add and modify verification rules according to business needs, use verification rules to verify code files and generate intuitive verification reports, so that users can deal with problem codes in a timely manner to ensure the correctness of the code.

[0035] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0036] Please refer to Figure 1 , Figure 1 This is a first flow chart of a code verification method for data governance provided by the present invention, the code verification method comprising:

[0037] S101. When it is detected that a code file is submitted to the verification system, an API interface is called to obtain file parameter information of the code file, and the code file is downloaded to a specified location according to the file parameter information.

[0038] S102. Perform multi-threaded verification on the code file according to pre-configured verification rules to determine problematic codes in the code file that do not comply with the verification rules. The verification rules include rules configured by the user in the database configuration table according to business requirements and / or verification rules developed through custom code. The database configuration table is a configuration table pre-created in the database for storing verification rules.

[0039] S103: Calling a professional knowledge base to provide code repair suggestions for the problem code, and generating a verification report including the problem code and the code repair suggestions.

[0040] In the construction and governance of modern data warehouses, database operations are a critical step. Irregular, erroneous, or low-quality SQL code can lead to program anomalies, data inconsistencies, performance degradation, and system crashes. Therefore, it is necessary to utilize validation rules to standardize and identify issues in SQL code. The code files in this article refer to the SQL code files used to manage and operate relational databases.

[0041] In order to cope with complex and ever-changing project requirements, it is necessary to flexibly customize verification rules according to different business needs. To this end, the present invention pre-creates a configuration table for storing verification rules in the database, and users can configure the required verification rules in the database configuration table according to business needs. For some specific industries or unique SQL coding specifications within the enterprise, users can also formulate verification rules by customizing the code. Based on this, when it is necessary to add, delete or modify verification rules, it can be achieved by operating the database configuration table or modifying the code to adapt to the ever-changing business needs and code specifications. The overall design is relatively flexible and has strong scalability.

[0042] After the verification rules are configured, when a code file is detected to be submitted to the verification system, the API (Application Programming Interface) interface is called to obtain the file parameter information of the code file (for example, the storage path of the code file, etc.), and the code file is downloaded to the specified location according to the file parameter information.

[0043] Detecting whether a code file has been submitted to the verification system can be achieved in the following ways: first, periodically polling the verification system at a preset time interval to detect whether a new code file has been submitted to the verification system; or by monitoring whether a code submission event has been received to determine whether a new code file has been submitted to the verification system. The code submission event is an event triggered when a user submits a code file; or, after the user submits the code file to the verification system, actively calling the verification system's designated API interface to trigger the verification system to detect whether a new code file has been submitted to the verification system. That is, the verification system can determine that a new code file has been submitted to the verification system after detecting that the designated API interface has been called by the user. The above various methods can be selected according to actual needs, and the present invention does not specifically limit this.

[0044] The code files are then checked item by item according to various pre-configured verification rules, and any code that does not comply with the verification rules is analyzed. These code is then identified as problematic code. Furthermore, the present invention supports multi-threaded verification of code files using pre-configured verification rules, speeding up code verification and enabling timely completion even of very large amounts of code to be checked. This ensures that it can handle large database projects or applications requiring frequent code checks.

[0045] For problem codes that do not comply with the verification rules, the present invention also uses the professional knowledge base to determine the code repair suggestions corresponding to the verification rules corresponding to the problem code, and writes the problem code and the code repair suggestions together into the verification report to generate an intuitive verification report. The content of the verification report is easy for users to understand, including both the problem code (including the location and type of the problem code) and the code repair suggestions for eliminating the problem code, so that users can promptly handle and correct the problem code and maintain the normal operation of the system.

[0046] In addition, after generating a verification report, it can be promptly sent to the client via email or corporate WeChat, allowing users to promptly fix problematic code and improve code quality. The verification report can also interact with the user. For example, users can click on a problem code in the verification report to view detailed information about the problem code, and can also choose to ignore problematic code detected by certain verification rules, improving the user experience.

[0047] In summary, the present invention provides a code verification method for data governance, including pre-configuring verification rules in the database configuration table according to business needs and / or formulating verification rules through custom codes, which can flexibly customize, add and modify verification rules according to business needs to meet the customized needs of financial and other industries for data security and compliance; after the code file is submitted to the verification system, the code file is multi-threadedly verified according to the pre-configured verification rules to determine the problem code in the code file that does not comply with the verification rules, thereby improving the code verification speed; calling the professional knowledge base to provide code repair suggestions for the problem code, generating a more intuitive verification report including the problem code and code repair suggestions, which is convenient for users to understand the report, quickly locate the problem code to ensure code quality, have good interactivity, reduce the difficulty of handling problems, and improve repair efficiency. Adapting to the future trend of data research and management, facing the growing special verification needs, it can quickly configure rules and complete verification, and continuously enable work efficiency improvement.

[0048] Based on the above embodiment:

[0049] The following describes some preprocessing steps before performing rule verification on the code file.

[0050] As an optional embodiment, after downloading the code file to the specified location according to the file parameter information, the method further includes:

[0051] Determine the file type of the code file and judge whether the file type of the code file is within the preset verification range; when it is determined that the code file is within the preset verification range, enter the step of performing multi-threaded verification on the code file according to pre-configured verification rules.

[0052] Please refer to Figure 2 , Figure 2 The second flow chart of a code verification method for data governance provided by the present invention. In this embodiment, after downloading the code file, the file type of the code file is first determined, such as different types of files such as shell script files, SQL code files, and table creation statements, and then it is determined whether the verification system supports the verification of the code file. It is specifically implemented by determining whether the file type of the code file is within the preset verification range. When it is determined that the code file is within the preset verification range, the step of performing multi-threaded verification on the code file according to the pre-configured verification rules is entered. For non-SQL code files that are uploaded by mistake, such as word documents, etc., they can be intercepted in advance to ensure the efficiency of code verification.

[0053] After downloading the code file to the specified location based on the file parameters, the code file is preprocessed to facilitate subsequent rule verification. This preprocessing includes standardizing the code file format and / or parsing the code file syntax. After preprocessing, the code file is subjected to multi-threaded verification according to the preconfigured verification rules.

[0054] The specific content of the rule verification in the present invention is described below.

[0055] As an optional embodiment, the verification rules include syntax and structure verification rules, naming standardization verification rules, security and compliance verification rules, performance optimization rules, maintainability and readability verification rules, and any one or more combinations of industry-specific rules.

[0056] The above-mentioned syntax and structure verification rules can detect basic syntax compliance and statement integrity; naming standardization verification rules can include object naming rules (for example, temporary tables must start with tmp_ or temp_) and prefix rules; security and compliance verification rules can include sensitive information verification, permission verification, and compliance verification; performance optimization rules can include complex operation detection, resource consumption detection, and index usage detection; maintainability and readability verification rules can include code format verification, comment coverage verification, etc.

[0057] The following describes the process of further analyzing and processing the verification report after generating the verification report including the problem code and code repair suggestions.

[0058] As an optional embodiment, after generating a verification report including problem codes and code repair suggestions, it also includes: aggregating and compressing the verification reports corresponding to each code file in the same batch, and sending them to the client to prompt the user to deal with the problem code in a timely manner.

[0059] Please refer to Figure 2 , Figure 2 The second flow chart of a code verification method for data governance provided by the present invention. The present invention aggregates and compresses the verification reports of the same batch and sends them to the client through various means such as corporate WeChat or email to ensure that users receive feedback in a timely manner, repair problem codes in a timely manner, and improve code quality. In addition, the verification report corresponding to each code file can be stored, and the verification reports corresponding to each code file of the same batch can be statistically analyzed to determine the frequency of each verification rule that verifies the problem code, and generate a visual statistical report. By performing visual data analysis, suggestions are given for concentrated problems to facilitate user rectification.

[0060] Please refer to Figure 3 , Figure 3This is a schematic diagram of the structure of a code verification system for data governance provided by the present invention, which includes:

[0061] The file download module 301 is used to call the API interface to obtain the file parameter information of the code file when detecting that the code file is submitted to the verification system, and download the code file to a specified location according to the file parameter information;

[0062] Verification module 302 is used to perform multi-threaded verification on the code file according to pre-configured verification rules, and identify problematic code in the code file that does not comply with the verification rules. The verification rules include rules configured by the user in the database configuration table according to business requirements and / or verification rules developed through custom code. The database configuration table is a configuration table pre-created in the database for storing verification rules;

[0063] The report generation module 303 is used to call the professional knowledge base to provide code repair suggestions for the problem code and generate a verification report including the problem code and the code repair suggestions.

[0064] For a detailed introduction to the code verification system for data governance provided by the present invention, please refer to the embodiment of the code verification method for data governance, and the present invention will not elaborate on it here.

[0065] Please refer to Figure 4 , Figure 4 A schematic diagram of a data governance-oriented code verification device provided by the present invention is provided. The data governance-oriented code verification device includes:

[0066] Memory 401, used for storing computer programs;

[0067] Processor 402 is configured to implement any of the steps of the aforementioned code verification method for data governance when executing a computer program.

[0068] For a detailed introduction to the code verification device for data governance provided by the present invention, please refer to the embodiment of the code verification method for data governance, and the present invention will not elaborate on it here.

[0069] The present invention also provides a storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the above-mentioned code verification methods for data governance.

[0070] For a detailed introduction to the storage medium provided by the present invention, please refer to the embodiment of the code verification method for data governance, which will not be elaborated here.

[0071] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. It should also be noted that in this specification, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. Moreover, the terms "comprises", "includes" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, article or device that includes a series of elements includes not only those elements, but also other elements that are not explicitly listed, or also includes elements that are inherent to such process, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a..." does not exclude the presence of other identical elements in the process, article or device that includes the element.

[0072] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not limited to the embodiments shown herein but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A code verification method for data governance, characterized in that: include: When a code file is detected to be submitted to the verification system, the API interface is called to obtain the file parameter information of the code file, and the code file is downloaded to a specified location according to the file parameter information; Performing multi-threaded verification on the code file according to pre-configured verification rules to determine problematic codes in the code file that do not comply with the verification rules, wherein the verification rules include rules configured by the user in a database configuration table according to business requirements and / or verification rules formulated through custom code, and the database configuration table is a configuration table pre-created in the database for storing the verification rules; The professional knowledge base is called upon to provide code repair suggestions for the problem code, and a verification report including the problem code and the code repair suggestions is generated.

2. The code verification method for data governance according to claim 1, characterized in that: After downloading the code file to the designated location according to the file parameter information, the method further includes: Determining the file type of the code file, and judging whether the file type of the code file is within a preset verification range; When it is determined that the code file is within the preset verification range, the step of performing multi-threaded verification on the code file according to pre-configured verification rules is entered.

3. The code verification method for data governance according to claim 1, characterized in that: After downloading the code file to the designated location according to the file parameter information, the method further includes: The code file is preprocessed, and after the preprocessing is completed, the step of performing multi-threaded verification on the code file according to pre-configured verification rules is entered, wherein the preprocessing includes code file format standardization and / or code file syntax parsing.

4. The code verification method for data governance according to claim 1, characterized in that: The detection code file is submitted to the verification system, including: Periodically poll the verification system at a preset time interval to detect whether there is a new code file submitted to the verification system, or determine whether there is a new code file submitted to the verification system by monitoring whether a code submission event is received, or determine whether there is a new code file submitted to the verification system after detecting that a specified API interface is called by a user, and enter the step of calling the API interface to obtain file parameter information of the code file, wherein the code submission event is an event triggered when the user submits a code file.

5. The code verification method for data governance according to claim 1, characterized in that: The verification rules include any one or more of the following: syntax and structure verification rules, naming standardization verification rules, security and compliance verification rules, performance optimization rules, maintainability and readability verification rules, and industry-specific rules.

6. The code verification method for data governance according to claim 1, characterized in that: After generating the verification report including the problem code and the code repair suggestion, the method further includes: The verification reports corresponding to each code file in the same batch are summarized and compressed, and sent to the client to prompt the user to deal with the problem code in time.

7. The code verification method for data governance according to claim 1, characterized in that: After generating the verification report including the problem code and the code repair suggestion, the method further includes: The verification report corresponding to each code file is stored, and the verification reports corresponding to each code file in the same batch are statistically analyzed to determine the frequency of each verification rule that verifies the problem code, and generate a visual statistical report.

8. A code verification system for data governance, characterized by: include: A file download module is used to call an API interface to obtain file parameter information of the code file when detecting that the code file is submitted to the verification system, and download the code file to a specified location according to the file parameter information; a verification module, configured to perform multi-threaded verification on the code file according to pre-configured verification rules, and determine problematic code in the code file that does not comply with the verification rules, wherein the verification rules include rules configured by the user in a database configuration table according to business requirements and / or verification rules formulated through custom code, and the database configuration table is a configuration table pre-created in the database for storing the verification rules; The report generation module is used to call the professional knowledge base to provide code repair suggestions for the problem code and generate a verification report including the problem code and the code repair suggestions.

9. A code verification device for data management, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the steps of the code verification method for data governance as described in any one of claims 1 to 7 when executing the computer program.

10. A storage medium, characterized in that: The storage medium stores a computer program, which, when executed by a processor, implements the steps of the code verification method for data governance as described in any one of claims 1 to 7.

Images