Method and system for acquiring total data structure, electronic equipment and medium

Abstract

The invention provides a full data structure acquisition method and system, electronic equipment and a medium, and the method comprises the steps: obtaining a to-be-analyzed system code, and scanning the to-be-analyzed system code to obtain a code intermediate table; performing semantic analysis on the code intermediate table to generate a semantic feature set and a data initial directory; grading each code block based on the semantic feature set, the data initial directory and a preset grading rule base, and generating an initial directory metadata table based on a grading result; and correcting the initial directory metadata table based on an existing system code directory structure in the database, and generating a data directory based on the corrected initial directory metadata table. According to the invention, the accuracy and comprehensiveness of data cataloguing are improved.

Description

Methods, systems, electronic devices, and media for acquiring full data structures Technical Field

[0001] This invention relates to the field of big data and data synchronization technology, and in particular to a method, system, electronic device and medium for acquiring full data structure. Background Technology

[0002] In the field of big data technology and data synchronization, data preliminary work is necessary to address the need for cross-departmental, cross-domain, and cross-level data synchronization. Currently, the core implementation path for data preliminary work is a combination of automated exploration and manual completion. Technical tools are used to automatically collect data metadata, which is then supplemented and improved manually to complete the data cataloging process. However, this solution has significant limitations in practical implementation: automated exploration requires direct access to the databases of data source units. Such operations easily infringe upon the data security and privacy protection boundaries of core business systems, leading to widespread resistance from units owning important core systems. Consequently, the application scope and actual effectiveness of automated exploration technology are severely limited.

[0003] As a result, current data cataloging work is still mainly done manually. However, manual operation lacks unified standards and effective verification mechanisms, which leads to problems with the completeness and accuracy of the final data catalog. The actual coverage and the true data situation are difficult to control precisely. Summary of the Invention

[0004] In view of this, the purpose of the present invention is to provide a method, system, electronic device and medium for acquiring full data structure, so as to improve the accuracy and comprehensiveness of data cataloging.

[0005] To achieve the above objectives, the technical solution adopted by the present invention is as follows: In a first aspect, the present invention provides a method for obtaining a full data structure, comprising: obtaining the code of the system to be analyzed and scanning the code to obtain an intermediate code table; performing semantic analysis on the intermediate code table to generate a semantic feature set and an initial data directory; classifying each code block based on the semantic feature set, the initial data directory and a preset hierarchical rule base, and generating an initial directory metadata table based on the hierarchical results; correcting the initial directory metadata table based on the existing system code directory structure in the database, and generating a data directory based on the corrected initial directory metadata table.

[0006] Optionally, the code of the system to be analyzed is scanned to obtain an intermediate code table, including: parsing the code of the system to be analyzed based on an AST parser and extracting the code structure information, comment text and code dependency relationship of the code of the system to be analyzed; based on the code structure information, comment text and code dependency relationship, deduplication and filtering are performed on the parsed system code to generate a structured intermediate code table; wherein, the intermediate code table includes at least: the file path of the code block, the code block identifier and the comment text.

[0007] Optionally, semantic analysis is performed on the intermediate code table to generate a semantic feature set and an initial data directory. This includes: performing natural language processing on the comment text of each code block in the intermediate code table to obtain the functional description of each code block; matching the code structure information with a pre-built code semantic feature library to determine the semantic label of each code block; normalizing the semantic labels and generating a semantic feature set based on the code block identifier, normalized semantic labels, and functional description of each code block; clustering the data whose semantic labels are in the data persistence layer, and dividing the clustered modules according to a preset level to generate an initial data directory.

[0008] Optionally, each code block is classified based on a semantic feature set, an initial data directory, and a preset hierarchical rule base. This includes: performing a two-layer scan on field names, enumeration values, and annotation text in the initial data directory based on a preset regular expression feature library and a large model semantic validator to generate an initial sensitive fingerprint vector; matching the initial sensitive fingerprint vector with the preset hierarchical rule base to obtain the initial level and corresponding rule number of each code block; verifying and correcting the initial level of the code block to obtain the target level of each code block, and writing the target level of the code block back to the initial data directory to generate an initial directory metadata table.

[0009] Optionally, the initial directory metadata table is corrected based on the existing system code directory structure in the database, including: obtaining the existing system code directory structure in the database and extracting the feature information of the system code directory structure; wherein, the feature information includes at least: directory hierarchy, module naming rules; the system code and the system code to be analyzed are of the same type; comparing the hierarchical result of the initial directory metadata table with the feature information of the system code directory structure; if the hierarchical result of the initial directory metadata table is different from the feature information of the system code directory structure, then the hierarchical result of the initial directory metadata table is corrected.

[0010] Optionally, a data directory is generated based on the corrected initial directory metadata table, including: generating a data directory based on the corrected initial directory metadata table and a preset directory template, and associating each directory node with a corresponding code block to obtain a code directory document.

[0011] Optionally, it also includes: monitoring the database in real time to obtain database change information; scanning the system code based on the change information to obtain the changed code, and performing semantic recognition on the changed code to obtain the semantic features of the changed code; generating a local hierarchical result of the changed code based on the semantic features of the changed code, and obtaining the updated data directory based on the local hierarchical result; comparing the database change information with the changed code; if the database change information is consistent with the changed code, comparing the updated data directory with the current data directory to generate a pre-directed data directory.

[0012] Secondly, the present invention provides a system for acquiring a full data structure, comprising: a code scanning module for acquiring the code of the system to be analyzed and scanning the code to obtain an intermediate code table; a semantic recognition module for performing semantic analysis on the intermediate code table to generate a semantic feature set and an initial data directory; an automatic classification module for classifying each code block based on the semantic feature set, the initial data directory, and a preset classification rule base, and generating an initial directory metadata table based on the classification results; and a directory generation module for correcting the initial directory metadata table based on the existing system code directory structure in the database, and generating a data directory based on the corrected initial directory metadata table.

[0013] Thirdly, the present invention provides an electronic device including a processor and a memory, the memory storing computer-executable instructions executable by the processor, the processor executing the computer-executable instructions to implement the steps of the method provided in any of the first aspects above.

[0014] Fourthly, the present invention provides a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, performs the steps of the method provided in any of the first aspects above.

[0015] The present invention brings the following beneficial effects: The method, system, electronic device, and medium for obtaining the above-mentioned full data structure provided by the present invention first acquire the system code to be analyzed and scan it to obtain an intermediate code table; then, semantic analysis is performed on the intermediate code table to generate a semantic feature set and an initial data directory; next, each code block is classified based on the semantic feature set, the initial data directory, and a preset hierarchical rule base, and an initial directory metadata table is generated based on the classification results; finally, the initial directory metadata table is corrected based on the existing system code directory structure in the database, and a data directory is generated based on the corrected initial directory metadata table. In the above method, the database table structure can be extracted by parsing system code files without accessing the production database, thereby ensuring data security; business semantic features are generated through semantic recognition, and each code block is classified according to a preset hierarchical rule base; finally, a data directory is generated after correction by comparing with the existing system code directory structure in the database, thereby improving the accuracy and comprehensiveness of data directory cataloging.

[0016] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention are realized and obtained in accordance with the structures particularly pointed out in the description, claims and drawings.

[0017] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0018] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0019] Figure 1 is a flowchart of a method for obtaining a full data structure according to an embodiment of the present invention; Figure 2 is a flowchart of a method for obtaining a full data structure that supports code scanning and database exploration according to an embodiment of the present invention; Figure 3 is a structural schematic diagram of a system for obtaining a full data structure according to an embodiment of the present invention; Figure 4 is a structural schematic diagram of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] Currently, in scenarios involving multi-departmental collaboration and heterogeneous, dispersed data, incomplete data cataloging has become a common pain point hindering data governance and value realization. Existing technical solutions face multiple severe challenges in promoting the implementation of full-scale data cataloging, consistently failing to achieve a balance between the three core requirements of "comprehensiveness, effectiveness, and timeliness" in data cataloging. Specific problems are as follows: On the one hand, existing cataloging methods have inherent defects such as incomplete coverage, low efficiency, and inconsistent quality. Automated exploration technology, due to its direct access to the databases of data source units, easily touches the data security boundaries of core business systems, resulting in limited application scope and difficulty in comprehensively exploring heterogeneous system data scattered across various departments, creating blind spots in data source coverage. On the other hand, manual cataloging is not only inefficient but also lacks unified standardized specifications, easily leading to problems such as duplicate data entry, omission of key data, and inconsistent cataloging quality, directly undermining the basic integrity and standardization of the data catalog.

[0022] On the other hand, there are significant bottlenecks in the collaboration and dynamic maintenance of the data catalog. The cross-departmental collaboration process is cumbersome and the division of responsibilities is unclear, making it difficult to synchronize data updates and severely delaying the dynamic maintenance of the catalog. More importantly, the lack of a unified dynamic verification mechanism makes it impossible to effectively verify the authenticity and accuracy of the cataloged data, ultimately resulting in a serious disconnect between the data catalog and the actual data sources, making it difficult to meet the actual needs of dynamic changes in government data and enterprise business data.

[0023] Based on this, the present invention provides a method, system, electronic device and medium for obtaining full data structure, which can improve the accuracy and comprehensiveness of data cataloging.

[0024] To facilitate understanding of this embodiment, a method for obtaining a full data structure disclosed in this embodiment of the invention will first be described in detail. This method can be executed by an electronic device, such as a smartphone, computer, or tablet computer. Referring to the flowchart of a method for obtaining a full data structure shown in Figure 1, the method mainly includes the following steps S101 to S104: Step S101: Obtain the code of the system to be analyzed and scan the code to obtain an intermediate code table.

[0025] In one implementation, for a system that requires data cataloging, the source code file of the system is read to obtain the code of the system to be analyzed, and then an Abstract Syntax Tree (AST) parser is used to deconstruct the source code to obtain an intermediate code table.

[0026] Step S102: Perform semantic analysis on the intermediate code table to generate a semantic feature set and an initial data directory.

[0027] In one implementation, after obtaining the intermediate code table, semantic analysis of the intermediate code table is performed using large model and retrieval augmentation generation (RAG) technology to generate a semantic feature set and an initial data directory.

[0028] Step S103: Classify each code block based on the semantic feature set, the initial data directory, and the preset hierarchical rule base, and generate an initial directory metadata table based on the classification results; In one implementation, the system's built-in regular feature library and large model semantic validator are called to perform a two-layer scan of the initial data directory to generate sensitive fingerprint vectors, and the sensitive fingerprint vectors are matched with the preset hierarchical rule base to obtain classification results; Then, after self-verification and correction of the classification results, the initial directory metadata table is generated.

[0029] Step S104: Correct the initial directory metadata table based on the existing system code directory structure in the database, and generate a data directory based on the corrected initial directory metadata table.

[0030] In one implementation, the system retrieves existing code directory structures of similar systems from the database and compares the retrieval results with the initial directory metadata table. If differences are found, corrections are performed, module names are adjusted, and hierarchical results are updated. Finally, a data directory is automatically generated based on the corrected initial directory metadata table.

[0031] The method for obtaining the full data structure provided in this embodiment of the invention can extract the database table structure by parsing system code files without accessing the production database, thereby ensuring data security; it generates business semantic features through semantic recognition, classifies each code block according to a preset hierarchical rule base, and finally generates a data directory by comparing it with the existing system code directory structure in the database, thereby improving the accuracy and comprehensiveness of data directory cataloging.

[0032] In one implementation, for the aforementioned step S101, that is, when scanning the code of the system to be analyzed to obtain the intermediate code table, the following methods may be used, including but not limited to: First, the code of the system to be analyzed is parsed based on the AST parser, and the code structure information, comment text and code dependency relationship of the code of the system to be analyzed are extracted.

[0033] In practice, the corresponding AST parser is called according to the file type of the system code to be analyzed (e.g., java, .py) to extract the code structure information (e.g., classes, functions, variables), comment text, and code dependencies of the system code to be analyzed.

[0034] In this embodiment of the invention, the AST parser can not only extract variable names from the code but also accurately identify ORM mapping relationships in the code (e.g., MyBatis's ResultMap, Hibernate's Entity annotation) and establish a hard-link mapping graph of class attributes and database fields. It can also automatically extract contextual logic related to data operations by traversing the call chain through the AST. For example, by analyzing the method names (e.g., `checkUserEligibility`) and input parameters in the `Service` layer, the business meaning of the data table fields in the underlying `DAO` layer can be inferred, solving the semantic loss problem caused by obscure database field names (e.g., `col_1`, `status_a`).

[0035] Then, based on code structure information, comment text, and code dependencies, the parsed system code is deduplicated and filtered to generate a structured intermediate code table.

[0036] In practice, the parsed code is deduplicated, redundant code (such as blank lines outside comments) is filtered out, and a structured intermediate code table is generated. The intermediate code table includes at least the file path of the code block, the code block identifier, and the comment text.

[0037] In one implementation, for the aforementioned step S102, that is, when performing semantic analysis on the intermediate code table to generate a semantic feature set and an initial data directory, the following methods may be used, including but not limited to: First, perform natural language processing on the comment text of each code block in the intermediate code table to obtain the functional description of each code block.

[0038] In practice, a large model is used to perform natural language processing (NLP) on the comment text of each code block to extract the core functional description of each code block (e.g., user information, vehicle registration certificate, etc.).

[0039] In this embodiment of the invention, a dedicated vector knowledge base for data governance can be constructed, utilizing retrieval-enhanced generation techniques to improve the illusion problem of large models. Specifically, the dedicated vector knowledge base for data governance includes pre-defined standard data element definitions (e.g., GB / T43697 standard, industry terminology lists, etc.). After extracting the core functional description of each code block, the extracted English terms (embedded vectorization) can be matched with the standard Chinese business terms in the knowledge base using vector similarity matching to ensure that the generated catalog names conform to industry standard specifications. Simultaneously, existing internal data dictionaries can be automatically retrieved, and for table structures with high similarity, confirmed business descriptions are automatically reused, achieving the accumulation and migration of catalog knowledge.

[0040] In one implementation, to address the common problem of lack of comments in dictionary codes (such as `status=1 / 2 / 3`) in databases, this embodiment of the invention employs Chain-of-Thought (CoT) hint engineering for intelligent reasoning of enumeration values. Specifically, the system automatically scans the entire code domain to locate the judgment condition of the field in the business logic (e.g., `ifstatus == 1 then return "Pending review"`). The LLM (Large Model) aggregates the logic fragments scattered throughout the code, infers the complete business meaning of the enumeration value (`1 pending review`, `2: approved`, `3: rejected`), and automatically generates data dictionary items to fill the gaps where the business meaning cannot be obtained solely from database metadata.

[0041] Then, the code structure information is matched with a pre-built code semantic feature library to determine the semantic label of each code block.

[0042] In practice, the code structure information (class / function name, parameter list) is combined with the code semantic feature library built in the database to mark each code block with a corresponding semantic tag, such as: authentication module, data persistence layer, etc.

[0043] Next, the semantic tags are normalized, and a semantic feature set is generated based on the code block identifier, normalized semantic tags, and functional description of each code block.

[0044] In practice, semantic tags are normalized (e.g., unified login verification is an authentication module) to generate a semantic feature set that includes code block identifiers, semantic tags, and function descriptions.

[0045] Finally, the data with semantic tags of the data persistence layer are clustered, and the clustered modules are divided according to a preset hierarchy to generate the initial data directory.

[0046] In practical implementation, for data with semantic tags in the data persistence layer, the modules are first clustered according to code complexity (number of functions, number of lines of code) and functional independence. Then, the clustered modules are divided into levels (e.g., first-level module: permission management, second-level sub-module: login authentication and role configuration, third-level sub-module: business interaction data) to generate the initial data directory.

[0047] In one implementation, for the aforementioned step S103, that is, when classifying each code block based on the semantic feature set, the initial data directory and the preset hierarchical rule base, the following methods can be adopted, including but not limited to: First, based on the preset regular feature base and the large model semantic verifier, the field names, enumeration values ​​and annotation text in the initial data directory are scanned in two layers to generate an initial sensitive fingerprint vector.

[0048] In practice, the system's built-in regular expression feature library and large model semantic validator are called to perform a two-layer scan on the field names, enumeration values, and comment texts of each code block in the initial data directory, generating an initial sensitive fingerprint vector that includes sensitive types, confidence levels, and field positions.

[0049] In this embodiment of the invention, a dual verification is performed by combining a regular expression feature library and NLP contextual understanding. This not only identifies explicit features (such as ID card number format), but also identifies implicit sensitive data by analyzing the variable names around the field (e.g., `param_sfz`), annotation risk levels, and contextual information. The confidence score of the sensitive data is calculated, and it is automatically labeled only when the score exceeds the threshold. The overall security level of the data table is automatically calculated according to the data classification rules (core / important / general).

[0050] Then, the initial sensitive fingerprint vector is matched with the preset hierarchical rule base to obtain the initial level and corresponding rule number of each code block.

[0051] In practice, the initial sensitive fingerprint vector is matched with a preset hierarchical rule base. For example, it is matched with the "core-important-general" three-level decision tree preset in the GB / T43697-2024 rule base to output the initial level and corresponding rule number of each code block.

[0052] Next, after verifying and correcting the initial level of the code block, the target level of each code block is obtained, and the target level of the code block is written back to the initial data directory to generate the initial directory metadata table.

[0053] In practice, the classification results of each code block are first self-verified. If the same logical table is assigned more than two levels in different code blocks, the conflict resolution engine is triggered. The table is automatically promoted according to the principle of choosing the higher level (i.e., the table level is determined according to the higher level. For example, if data A is assigned level 1 in code block 1 and level 4 in code block 2, and level 4 is higher than level 1, then the level of data A is unified to level 4) and written to the exception log.

[0054] After self-verifying the grading results, security operations personnel can also adjust the grade determined by each code block or add custom rules in the graphical interface. The system re-runs the decision tree in real time to generate new grades and highlights the grading differences. It also supports one-click adoption or rollback.

[0055] After the security operations personnel correct the classification results, a classification confirmation form with a digital signature is generated, the final level (i.e., the target level) is locked, and the target level is written back to the initial data directory to generate the initial directory metadata table. At the same time, an interpretable audit trail is output for subsequent compliance review and traceability.

[0056] In one implementation, for the aforementioned step S04, i.e., when correcting the initial directory metadata table based on the existing system code directory structure in the database, the following methods may be used, including but not limited to: First, obtain the existing system code directory structure in the database and extract the feature information of the system code directory structure; wherein, the feature information includes at least: directory hierarchy, module naming rules; the system code and the system code to be analyzed are of the same type; then, compare the hierarchical result of the initial directory metadata table with the feature information of the system code directory structure; if the hierarchical result of the initial directory metadata table is different from the feature information of the system code directory structure, then correct the hierarchical result of the initial directory metadata table.

[0057] In practice, after obtaining the initial directory metadata table, a one-time exploration and directory comparison correction are performed based on the database. First, the existing system code directory structure with the same code type as the system code to be analyzed is retrieved from the database, and feature information such as directory hierarchy and module naming rules is extracted. Then, the hierarchical results in the initial directory metadata table generated in the previous steps are compared with the existing directories in the database (e.g., comparing module name consistency, hierarchy rationality, etc.). If differences are found (e.g., permission management in the hierarchical results is set to user permissions in the existing directory), correction is performed: the module names are adjusted to the common names of the existing directories, and the hierarchical results are updated synchronously.

[0058] In one implementation, for the aforementioned step S04, that is, when generating a data directory based on the corrected initial directory metadata table, the following methods may be used, including but not limited to: generating a data directory based on the corrected initial directory metadata table and a preset directory template, and associating each directory node with a corresponding code block to obtain a code directory document.

[0059] In practice, following the preset directory template (e.g., first-level module → second-level sub-module → function point), the source of the directory is attached to the data directory, and each directory node is associated with a corresponding code block (including file path and code block identifier) ​​to generate a directory-code mapping relationship; finally, the code directory document (e.g., a tree-structured markdown document) is output, and the directory generation status is recorded (e.g., XX-level directory generated, containing XX modules).

[0060] In one implementation, to ensure the real-time nature of the data catalog, the method further includes: first, monitoring the database in real time to obtain database change information. In specific implementations, the database is monitored in real time, and database change information (such as added, modified, or deleted fields) is captured and probed synchronously.

[0061] Then, based on the change information, the system code is scanned to obtain the changed code, and the semantic features of the changed code are obtained by semantic recognition.

[0062] Next, based on the semantic features of the change code, a local hierarchical result of the change code is generated, and based on the local hierarchical result, the updated data directory is obtained.

[0063] In practice, when a change in the data table structure in the database is detected, code scanning is triggered to obtain the changed code, and the semantic features of the changed code are extracted again. The local hierarchy is then re-executed to obtain the local hierarchy results, and the updated data directory is obtained based on the local hierarchy results.

[0064] Finally, the database change information is compared with the change code; if the database change information and the change code are consistent, the updated data directory is compared with the current data directory to generate a pre-directed data directory.

[0065] In practice, the change information in the database is compared with the change results of the code scan to ensure consistency of sources; at the same time, the updated data directory is compared with the current data directory to generate a pre-directed data directory to maintain real-time consistency between the directory, code and database.

[0066] In this embodiment of the invention, by directly connecting to the data source, dynamic monitoring and structural verification of the cataloged data directory are implemented. This allows for real-time capture of metadata changes such as table additions / deletions and structural alterations, automatically updating the directory synchronously. Furthermore, by deeply examining the data table structure, field types, and constraints, a consistency comparison is performed with the cataloged metadata to identify discrepancies and issue alerts. Simultaneously, a data quality rule base can be integrated to automatically assess data integrity, accuracy, and standardization, and to trace the source of problems.

[0067] This invention employs a triple hybrid architecture of "AST structure parsing + contextual semantic derivation + domain knowledge enhancement" to achieve high-precision automated cataloging of data assets. Based on this, this invention provides a method for obtaining the full data structure by supporting code scanning and database exploration, as shown in Figure 2, including the following steps: Step 0: System preparation and database initialization.

[0068] Specifically, the deployment of the code analysis system includes the initialization of the code scanning module, semantic recognition module, automatic classification module, and directory generation module; initialization of the database, establishment of the code semantic feature library, classification rule library, and directory template library, and configuration of database connection parameters; and the configuration of code classification rules (such as by functional module, complexity, etc.) in the automatic classification module and the configuration of directory structure source templates (such as by module-submodule-functional point hierarchy) in the automatic directory generation module.

[0069] Step 1: System code scanning and preprocessing.

[0070] Specifically, the code scanning module reads the system code to be analyzed and performs full code parsing, including: ① calling the corresponding parser according to the code file type (e.g., .java, .py) to extract the code's structural information (classes, functions, variables), comment content, and code dependencies; ② deduplicating and filtering redundant code (e.g., blank lines outside comments) from the parsed code, and generating a structured intermediate code table (including file paths, code block identifiers, and comment text); ③ synchronizing the preprocessed intermediate code table to the semantic recognition module, while recording the code scanning status (e.g., "XX files have been scanned, and XX code blocks have been successfully parsed").

[0071] Step 2: Code semantic recognition and feature extraction.

[0072] Specifically, the semantic recognition module receives the intermediate code table and performs semantic analysis, including: ① performing natural language processing on the comment text of the code block to extract the core function description; ② combining the code structure information with the semantic feature library in the database to label each code block with a semantic tag; ③ normalizing the semantic tags (e.g., unifying "login verification" as "authentication module") to generate a semantic feature set containing "code block identifier - semantic tag - function description", which is then synchronized to the security automatic classification module.

[0073] ④ Based on code complexity (number of functions, number of lines of code) and functional independence, the clustered modules are divided into hierarchical sub-modules (e.g., first-level module: permission management, second-level sub-module: login authentication and role configuration, third-level sub-module: business interaction data) to generate an initial data directory.

[0074] Step 3: Automatic classification and grading of code module security.

[0075] Specifically, the code module security is automatically classified and graded. After the initial data directory is entered, the system automatically completes the grade confirmation in the following sub-steps, and the whole process is auditable and traceable: ① Automatic extraction of sensitive features: The built-in regular feature library and large model semantic verifier are called to perform a two-layer scan on the field names, enumeration values ​​and comment texts involved in each code block in the directory, and generate an initial sensitive fingerprint vector containing sensitive type, confidence level and field position.

[0076] ② Initial labeling of sensitive data: Match the sensitive fingerprint vector with the "core-important-general" three-level decision tree pre-set in the GB / T43697-2024 rule base, and output the initial level and corresponding rule number of each code block.

[0077] ③ Self-verification of hierarchical results: If the same logical table is assigned more than two levels in different code blocks, the conflict resolution engine is triggered, and the higher level is automatically promoted according to the principle of "choosing the higher level" and written to the exception log.

[0078] ④ Correction of classification results: Security operations personnel can adjust the classification or add custom rules in the graphical interface. The system will rerun the decision tree in real time and highlight the differences, supporting one-click adoption or rollback.

[0079] ⑤ Confirmation of grading results: After the correction is completed, a grading confirmation form with a digital signature is generated, the final grade is locked, it is written back to the initial directory metadata table, and an interpretable audit link is output for subsequent compliance review and traceability.

[0080] Step 4: One-time database exploration and directory comparison and correction.

[0081] Specifically, the database one-time exploration module is activated to perform a one-time exploration and directory comparison and correction, including: ① retrieving the existing directory structure of similar system code in the database and extracting features such as directory hierarchy and module naming rules; ② comparing the module classification results generated by the automatic classification module with the existing directories in the database (such as consistency of module names and rationality of hierarchy); ③ if differences are found, performing correction: adjusting the module names to the common naming of the existing directories, and synchronously updating the classification results to the automatic directory generation module.

[0082] Step 5: Automatically generate the code directory.

[0083] Specifically, the automatic directory generation module receives the corrected hierarchical results and performs directory generation: ① According to the preset directory template (such as first-level module → second-level sub-module → function point), the source of the directory is attached to the data directory; ② Associate each directory node with the corresponding code block (including file path and code block identifier) ​​to generate a directory-code mapping relationship; ③ Output the final code directory document (such as a tree-structured markdown document) and record the directory generation status (such as "Generated XX-level directory, containing XX modules").

[0084] Step 6: Continuous database exploration and directory synchronization updates.

[0085] Specifically, the continuous database probing module is activated to achieve continuous probing and synchronized directory updates, including: ① Real-time monitoring of the database to synchronously capture probing updates (such as added, modified, or deleted fields); ② When the data table structure changes, code scanning is triggered, the semantic recognition module re-extracts the semantic features of the changed code, and the automatic classification module re-executes local classification; ③ Comparing the database change and code scan change results to ensure consistency of sources; ④ Comparing the updated data directory with the current directory to generate a pre-compiled data directory, maintaining real-time consistency between the directory, code, and database.

[0086] The method for obtaining full data structure through code scanning and database exploration provided in this invention uses a non-intrusive code scanning module to extract database table structure and relationships by parsing source code files without accessing the production database; a semantic recognition module performs natural language processing on code comments and enumeration values ​​to automatically generate business semantic tags; a national standard rule engine automatically outputs data classification results conforming to GB / T 43697-2024 based on sensitive field characteristics; incremental DDL monitoring captures table structure change events in the version repository or test repository in real time and triggers directory synchronization updates; and bidirectional verification compares the differences between the directory and the actual database structure using hash fingerprints and generates traceable anomaly reports. This invention achieves full, real-time, and reliable data cataloging in a security-constrained environment through three-dimensional collaboration of source code reverse parsing, semantic completion, and incremental synchronous verification, improving the accuracy and comprehensiveness of data cataloging.

[0087] In addition to the full data structure acquisition method provided in the foregoing embodiments, this invention also provides a full data structure acquisition system. Referring to Figure 3, a schematic diagram of a full data structure acquisition system is shown, indicating that the system mainly includes the following parts: a code scanning module 301, used to acquire the code of the system to be analyzed and scan the code of the system to be analyzed to obtain a code intermediate table.

[0088] The semantic recognition module 302 is used to perform semantic analysis on the intermediate code table and generate a semantic feature set and an initial data directory.

[0089] The automatic classification module 303 is used to classify each code block based on the semantic feature set, the initial data directory and the preset classification rule base, and generate an initial directory metadata table based on the classification results.

[0090] The directory generation module 304 is used to correct the initial directory metadata table based on the existing system code directory structure in the database, and generate a data directory based on the corrected initial directory metadata table.

[0091] The full data structure acquisition system provided in this embodiment of the invention can extract the database table structure by parsing system code files without accessing the production database, thereby ensuring data security; it generates business semantic features through semantic recognition, classifies each code block according to a preset hierarchical rule base, and finally generates a data directory after correction by comparing with the existing system code directory structure in the database, thereby improving the accuracy and comprehensiveness of data directory cataloging.

[0092] In one embodiment, the code scanning module 301 is specifically used to: parse the system code to be analyzed based on the AST parser, and extract the code structure information, comment text and code dependency relationship of the system code to be analyzed; based on the code structure information, comment text and code dependency relationship, perform deduplication and filtering on the parsed system code to generate a structured code intermediate table; wherein, the code intermediate table includes at least: the file path of the code block, the code block identifier and the comment text.

[0093] In one implementation, the semantic recognition module 302 is specifically used for: performing natural language processing on the annotation text of each code block in the intermediate code table to obtain the functional description of each code block; matching the code structure information with a pre-built code semantic feature library to determine the semantic label of each code block; normalizing the semantic label and generating a semantic feature set based on the code block identifier, normalized semantic label and functional description of each code block; clustering the data whose semantic label is the data persistence layer, and dividing the clustered modules according to a preset level to generate an initial data directory.

[0094] In one implementation, the automatic classification module 303 is specifically used to: perform a two-layer scan on field names, enumeration values, and annotation text in the initial data directory based on a preset regular feature library and a large model semantic validator to generate an initial sensitive fingerprint vector; match the initial sensitive fingerprint vector with a preset classification rule library to obtain the initial level and corresponding rule number of each code block; after verifying and correcting the initial level of the code block, obtain the target level of each code block, and write the target level of the code block back to the initial data directory to generate an initial directory metadata table.

[0095] In one embodiment, the directory generation module 304 is specifically used to: obtain the directory structure of existing system code in the database, and extract the feature information of the directory structure of the system code; wherein, the feature information includes at least: directory hierarchy, module naming rules; the system code and the system code to be analyzed are of the same type; compare the hierarchical result of the initial directory metadata table with the feature information of the directory structure of the system code; if the hierarchical result of the initial directory metadata table is different from the feature information of the directory structure of the system code, then correct the hierarchical result of the initial directory metadata table.

[0096] In one embodiment, the directory generation module 304 is specifically used to: generate a data directory based on the corrected initial directory metadata table and the preset directory template, and associate each directory node with a corresponding code block to obtain a code directory document.

[0097] In one embodiment, the system further includes: a database probing module, used to monitor the database in real time and obtain database change information; based on the change information, scan the system code to obtain changed code, and perform semantic recognition on the changed code to obtain semantic features of the changed code; based on the semantic features of the changed code, generate a local hierarchical result of the changed code, and based on the local hierarchical result, obtain an updated data directory; compare the database change information with the changed code; if the database change information is consistent with the changed code, compare the updated data directory with the current data directory to generate a pre-directed data directory.

[0098] It should be noted that the system provided in this embodiment of the invention has the same implementation principle and technical effects as the aforementioned method embodiment. For the sake of brevity, any parts not mentioned in the system embodiment can be referred to the corresponding content in the aforementioned method embodiment. The specific numerical values ​​provided in the implementation of this invention are merely exemplary and are not intended to limit the scope of the invention.

[0099] This invention also provides an electronic device, specifically, the electronic device includes a processor and a storage device; the storage device stores a computer program, and the computer program, when run by the processor, executes the method described in any of the above embodiments.

[0100] Figure 4 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. The electronic device 100 includes: a processor 40, a memory 41, a bus 42 and a communication interface 43. The processor 40, the communication interface 43 and the memory 41 are connected through the bus 42. The processor 40 is used to execute executable modules, such as computer programs, stored in the memory 41.

[0101] The memory 41 may include high-speed random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Communication between this system network element and at least one other network element is achieved through at least one communication interface 43 (which can be wired or wireless), such as the Internet, wide area network, local area network, metropolitan area network, etc.

[0102] Bus 42 can be an ISA bus, PCI bus, or EISA bus, etc. The bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only a single bidirectional arrow is used in Figure 4, but this does not indicate that there is only one bus or one type of bus.

[0103] The memory 41 is used to store programs. After receiving an execution instruction, the processor 40 executes the program. The method executed by the device for defining the flow process disclosed in any of the foregoing embodiments of the present invention can be applied to the processor 40 or implemented by the processor 40.

[0104] Processor 40 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of processor 40 or by instructions in software form. Processor 40 can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this invention. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this invention can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules can reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory 41. The processor 40 reads the information in memory 41 and, in conjunction with its hardware, completes the steps of the above method.

[0105] The computer program product of the readable storage medium provided in the embodiments of the present invention includes a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the methods described in the foregoing method embodiments. For specific implementation, please refer to the foregoing method embodiments, which will not be repeated here.

[0106] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0107] Finally, it should be noted that the above-described embodiments are merely specific implementations of the present invention, used to illustrate the technical solutions of the present invention, and not to limit it. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed in the present invention, or make equivalent substitutions for some of the technical features; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for obtaining a full data structure, characterized in that, include: Obtain the code of the system to be analyzed, and scan the code to obtain an intermediate code table; Semantic analysis is performed on the intermediate code table to generate a semantic feature set and an initial data directory; Each code block is classified based on the semantic feature set, the initial data directory, and the preset hierarchical rule base, and an initial directory metadata table is generated based on the classification results. The initial directory metadata table is corrected based on the existing system code directory structure in the database, and a data directory is generated based on the corrected initial directory metadata table.

2. The method according to claim 1, characterized in that, The process of scanning the code of the system to be analyzed to obtain an intermediate code table includes: parsing the code of the system to be analyzed using an AST parser and extracting the code structure information, comment text, and code dependencies of the code of the system to be analyzed; based on the code structure information, the comment text, and the code dependencies, deduplication and filtering are performed on the parsed system code to generate a structured intermediate code table; wherein, the intermediate code table includes at least: the file path of the code block, the code block identifier, and the comment text.

3. The method according to claim 2, characterized in that, Semantic analysis is performed on the intermediate code table to generate a semantic feature set and an initial data directory. This includes: performing natural language processing on the comment text of each code block in the intermediate code table to obtain a functional description of each code block; matching the code structure information with a pre-built code semantic feature library to determine the semantic tag of each code block; normalizing the semantic tags and generating a semantic feature set based on the code block identifier, normalized semantic tags, and functional description of each code block; clustering the data whose semantic tags are in the data persistence layer, and dividing the clustered modules according to a preset level to generate an initial data directory.

4. The method according to claim 1, characterized in that, The code block is classified based on the semantic feature set, the initial data directory, and the preset hierarchical rule base, including: performing a two-layer scan on the field names, enumeration values, and annotation text in the initial data directory based on the preset regular expression feature library and large model semantic validator to generate an initial sensitive fingerprint vector; matching the initial sensitive fingerprint vector with the preset hierarchical rule base to obtain the initial level and corresponding rule number of each code block; verifying and correcting the initial level of the code block to obtain the target level of each code block, and writing the target level of the code block back to the initial data directory to generate an initial directory metadata table.

5. The method according to claim 1, characterized in that, The initial directory metadata table is corrected based on the existing system code directory structure in the database, including: obtaining the existing system code directory structure in the database and extracting the feature information of the system code directory structure; wherein, the feature information includes at least: directory hierarchy and module naming rules; the system code is of the same type as the system code to be analyzed; comparing the hierarchical result of the initial directory metadata table with the feature information of the system code directory structure; if the hierarchical result of the initial directory metadata table is different from the feature information of the system code directory structure, then the hierarchical result of the initial directory metadata table is corrected.

6. The method according to claim 5, characterized in that, The process of generating a data directory based on the corrected initial directory metadata table includes: generating a data directory based on the corrected initial directory metadata table and a preset directory template, and associating each directory node with a corresponding code block to obtain a code directory document.

7. The method according to claim 1, characterized in that, Also includes: The database is monitored in real time to obtain change information. Based on the change information, the system code is scanned to obtain the changed code, and the semantic features of the changed code are obtained by semantic recognition. Based on the semantic features of the change code, a local hierarchical result of the change code is generated, and based on the local hierarchical result, the updated data directory is obtained; Compare the change information in the database with the change code; If the database change information matches the change code, the updated data directory is compared with the current data directory to generate a pre-directed data directory.

8. A system for acquiring a full data structure, characterized in that, include: The code scanning module is used to acquire the code of the system to be analyzed and to scan the code of the system to be analyzed to obtain an intermediate code table; The semantic recognition module is used to perform semantic analysis on the code intermediate table and generate a semantic feature set and an initial data directory; An automatic classification module is used to classify each code block based on the semantic feature set, the initial data directory, and a preset classification rule base, and to generate an initial directory metadata table based on the classification results. The directory generation module is used to correct the initial directory metadata table based on the existing system code directory structure in the database, and generate a data directory based on the corrected initial directory metadata table.

9. An electronic device, characterized in that, The method includes a processor and a memory, the memory storing computer-executable instructions executable by the processor, the processor executing the computer-executable instructions to implement the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program thereon, characterized in that, The computer program is executed by the processor to perform the steps of the method described in any one of claims 1 to 7.

Images