Efficient code feature extraction method suitable for different programming languages

Abstract

The invention discloses an efficient code feature extraction method suitable for different programming languages, which comprises the following steps: loading different programming language configuration files from a specified project path, analyzing the language configuration files through a predefined language analysis rule mapper, and generating a language rule mapping table suitable for different programming languages; identifying programming language types, creating a multi-task queue according to the types and distributing task threads; preprocessing the source code according to the language rule mapping table, performing word segmentation and marking, and normalizing a word segmentation result to generate a normalized source code file; performing code block segmentation on the normalized source code file according to a given window and step size, generating hash values as characteristic values, and obtaining a code file taking a hash value sequence as a characteristic fingerprint; and carrying out serialized storage and output on the code file taking the hash value sequence as the feature fingerprint and the marked content according to a standardized format. According to the method, the code analysis efficiency and accuracy can be improved.

Description

Technical Field

[0001] The invention relates to an efficient code feature extraction method applicable to different programming languages, belonging to the technical field of intelligent software analysis and processing. Background Art

[0002] In modern software engineering, code feature extraction plays an important role in source code analysis, similarity detection, code clone detection, etc. With the continuous increase in software scale and complexity, as well as the increase in the types of programming languages, code feature extraction methods face many challenges.

[0003] Traditional code feature extraction methods usually rely on the lexical unit and grammatical structure of the code. The analysis of lexical units is a common feature representation method in natural language processing and can also be applied to code. In the code, the lexical analysis model represents the code snippet as a collection of words (such as identifiers, keywords, operators, etc.) that appear in it, ignoring their order and grammatical structure. The defect of the lexical unit analysis method is that it cannot capture the grammatical and structural information of the code and cannot effectively handle code rearrangement or format changes. The method based on grammatical structure generally uses a grammatical analyzer to convert the code into an abstract syntax tree, and then extracts the structure and node information of the abstract syntax tree as features. Its defect is that it is easily restricted by the syntax and structure of a specific programming language, the differences between different languages ​​and code styles are large, and the construction and analysis of the syntax tree are complex. When many existing methods process source code in multiple programming languages, each new language needs to be added to the software to develop a parser for the corresponding language, and the source code needs to be added and modified. The scalability to the new language is poor, and it is impossible to use a general parser + configuration method for non-coding expansion, which is time-consuming and labor-intensive, and prone to omissions and errors.

[0004] In addition, existing open source tools for code feature extraction often lack structured design, resulting in low processing efficiency and inability to meet the needs of large-scale code analysis. For projects with large code sizes, traditional code feature extraction methods often lack structured design: serial traversal of each project file for preprocessing, or loading the full amount of project code for syntax tree analysis. These methods have low processing efficiency and may take a lot of time. For example, the open source tool sourcerCC that uses lexical analysis needs to preprocess and clean the code before extracting code features. This process is to serially traverse each project file to remove comments, format the code, etc. Especially when processing a large number of code files, these preprocessing steps may take a lot of time. Methods based on syntax analysis trees (ASTs) may require deep traversal and parsing of the code structure. If the code structure is complex or deeply nested, it may cause the execution time of the algorithm to increase significantly. Summary of the invention

[0005] The technical problem solved by the present invention is: to overcome the deficiencies of the prior art, to provide an efficient code feature extraction method suitable for different programming languages, to provide an efficient code feature extraction solution for multi-language projects, and to improve the efficiency and accuracy of code analysis.

[0006] The technical solution of the present invention is: an efficient code feature extraction method applicable to different programming languages, comprising:

[0007] Load different programming language configuration files supported by this project from the specified project path, parse the corresponding programming language configuration files through the predefined language parsing rule mapper, and generate language rule mapping tables applicable to different programming languages;

[0008] Identify the programming language type, create a multi-task queue and assign task threads according to the programming language type, pre-process the source code according to the language rule mapping table, perform word segmentation and tagging, normalize the word segmentation results and generate a normalized source code file;

[0009] The normalized source code file is segmented into code blocks according to the given window and step size and a hash value is generated as a feature value to obtain a code file with a hash value sequence as a feature fingerprint;

[0010] The code files and tag contents with hash value sequences as feature fingerprints are serialized, stored and output in a standardized format.

[0011] Preferably, the programming language configuration file defines the grammatical rules and feature extraction rules of the programming language; currently, the configuration files are mainly divided into two types:

[0012] One is the formulation of rules about programming language grammar, which describes in detail the grammatical structure, keywords, operators and other language features of the language;

[0013] The other is about the feature extraction rules of the language, which describes in detail how to identify and extract key features in the code.

[0014] Preferably, when identifying the programming language type, the programming language type is determined by identifying the suffix of the source code file.

[0015] Preferably, when creating a multi-task queue and allocating task threads according to programming language types, one thread is allocated to each programming language to process the file matching task of the corresponding language.

[0016] Preferably, the process of preprocessing the source code is as follows:

[0017] Set specific identifiers according to the programming language type;

[0018] Read the comment rules, literal rules, and reserved word list in the language rule mapping table, and initialize the relevant variables according to the corresponding rules;

[0019] Read the source code content and convert it to a string.

[0020] Preferably, the character string generated by preprocessing is used in the word segmentation and tagging process, and the character string generated by preprocessing is traversed character by character to perform word segmentation and tagging;

[0021] The marked content is: during the word segmentation process, the row and column position and processing type of the processed content before and after each word segmentation process.

[0022] Preferably, the specific processing content of word segmentation is as follows:

[0023] Whitespace characters: Whitespace characters are ignored, but the whitespace characters at the beginning of the line are counted as indents; if spaces or tabs are used, they are deleted, and the number of spaces or tabs is counted and recorded; the line and column numbers before and after the deletion are used as markers;

[0024] Indentation: Process the start and end positions of the code block according to the indentation stack and identify the code block structure; delete the indentation and use the line and column numbers before and after the deletion as markers;

[0025] Comments: Identify and skip single-line and multi-line comments according to the comment rules in the language rule mapping table by matching the comment start and end markers; for multi-line comments, identify nested comments; delete comments, and use the line and column numbers before and after the deletion as markers;

[0026] Literals: Identify and process literals according to literal rules, using the row and column numbers of the literals as markers;

[0027] Identifiers and numbers: Identifiers and numbers are matched by regular expressions, and the row and column numbers of the identifiers and numbers are used as markers;

[0028] Symbol: The present invention will directly use a single symbol as a mark, and directly use the row number and column number where the single symbol is located as the mark.

[0029] Preferably, the normalization of the word segmentation results is specifically as follows:

[0030] Normalize the numbers after word segmentation to the same digital type value;

[0031] Normalize the segmented strings into the same string type value;

[0032] Normalize the variables after word segmentation to the same variable type value;

[0033] The keywords after segmentation are not normalized and the original values ​​are kept;

[0034] Normalize the punctuation after word segmentation to the same identifier type value.

[0035] Preferably, the method for obtaining a code file with a hash value sequence as a characteristic fingerprint is specifically as follows:

[0036] S3-1 determines the window size and step size for feature extraction;

[0037] S3-2 splits a code block from the normalized source code file according to the window;

[0038] S3-3 generates a hash value of the code block as a characteristic value of the code block and records it;

[0039] S3-4 moves the feature extraction window by the step length, returns to step S3-2 and continues, traverses all the codes, and obtains a code file with a hash value sequence as a feature fingerprint.

[0040] Compared with the prior art, the present invention has the following advantages:

[0041] (1) The present invention parses the source code of multiple programming languages ​​and extracts its characteristic fingerprints, thereby realizing efficient analysis and processing of the code;

[0042] (2) The present invention divides the code file collections of different languages ​​in the project into several groups, creates independent threads for each language type, and uses a thread pool unified scheduling management method to greatly improve the processing capability and efficiency of the present method, reducing manual intervention and potential errors;

[0043] (3) The present invention serializes and stores the feature fingerprint data and counts additional information such as the number of lines in the source code file, thereby providing detailed data support for subsequent analysis and processing. This function not only facilitates the management and use of data, but also helps developers and testers better understand and analyze the source code, thereby improving the quality and efficiency of software development and testing. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 It is a schematic diagram of the overall process of the method of the present invention;

[0045] Figure 2 A schematic diagram of multi-threaded processing for multi-language files in the method of the present invention;

[0046] Figure 3 This is a schematic diagram of normalizing variables in the present invention. DETAILED DESCRIPTION

[0047] The present invention proposes an efficient code feature extraction method applicable to different programming languages. This method should be highly flexible and scalable, and can be optimized according to the characteristics of different programming languages. At the same time, the method should have efficient processing capabilities and be able to quickly and accurately extract code features to meet the needs of large-scale code analysis.

[0048] The method of the present invention proposes a configuration file definition in a unified format according to the characteristics of each language, and reads and parses the configuration file through a series of flexible configuration file loads and uses a corresponding parser to convert it into an internal data structure or object for subsequent file preprocessing and code feature extraction.

[0049] At the same time, in order to improve the processing efficiency, for the items to be extracted, the method of the present invention obtains the file set and distinguishes the language types, and adopts a multi-threaded processing mechanism for multi-language files, making full use of the capabilities and resources of multi-core processors to significantly improve the processing efficiency of code feature extraction.

[0050] The following is a further detailed description of a source code feature fingerprint extraction method applicable to multiple programming languages ​​provided in an embodiment of the present application in conjunction with the accompanying drawings. The specific implementation of the method may include the following steps: Figure 1 As shown:

[0051] Step 1: Programming language configuration file loading phase.

[0052] The multi-language configuration file loading mechanism proposed by the present invention is one of the key steps to realize multi-language code feature extraction. Firstly, the present invention generates a language rule mapping table applicable to different programming languages ​​by loading different programming language configuration files from a specified project path.

[0053] Specifically, when the present invention is started, it will automatically load different programming language configuration files supported by the project from the specified project path, and parse the language configuration files through the predefined language parsing rule mapper. The configuration file of each language mainly defines the grammatical rules and feature extraction rules of the programming language in detail. The current configuration files are mainly divided into two types: one is the rule formulation about the grammar of the programming language, which describes the grammatical structure, keywords, operators and other language characteristics of the language in detail; the other is the feature extraction rules about the language, which describe in detail how to identify and extract key features in the code. For example, feature extraction rules may include identifying specific grammatical patterns, code block structures, function definitions and calling patterns, etc. This division ensures that the present invention can accurately parse and process source code of different programming languages, extract meaningful feature fingerprints, and thus support subsequent code analysis and processing.

[0054] Table 1 Description of the contents of configuration files for different programming languages

[0055]

[0056] By parsing these configuration files through the language parsing rule mapper, corresponding language rule mapping tables can be generated. These mapping tables will be used in conjunction with the project source code files in the source code file preprocessing stage to ensure that the source code file preprocessing stage can accurately segment and normalize the code file contents of different programming languages. This configuration file loading mechanism not only enables the present invention to flexibly support multiple programming languages, but also allows dynamic updating and expansion of new language support as needed.

[0057] At present, we have realized the support of the following 28 programming languages: C, COBOL, C++, C#, Go, Haskell, Java, Python, Perl, PHP, Ruby, Scala, ST, VBA, Erlang, Verilog, VHDL, Fortran, Groovy, JavaScript, Kotlin, Lua, Objective-C, R, Rust, Swift, SystemVerilog, TypeScript. Through this design, the present invention can adapt to the complex environment of multi-language coexistence in modern software development, and provide strong support for code analysis and processing of multi-language projects.

[0058] Step 2: Source code file preprocessing, word segmentation and tagging stage.

[0059] In this step, the input project source code file is preprocessed in combination with the language rule mapping tables of different programming languages ​​in step 1 to facilitate feature fingerprint extraction and analysis in the source code feature fingerprint extraction stage. The preprocessing stage is an important part of the present invention, which ensures that the source code can be processed and analyzed in a unified and standardized format. When this stage is initialized, the language rule mapping table, programming language type, source code file, file path and character set are accepted as parameters.

[0060] First, by identifying the suffix of the source code file, the programming language is identified according to the suffix of the file, and the programming language type is determined, as shown in Table 2. A multi-task queue is created according to the programming language type, and task threads are allocated, as shown in Table 2. Figure 2 As shown in the figure. Each programming language can be assigned a thread to handle the file matching task of the corresponding language. Multi-threaded processing according to the language type can effectively improve the efficiency of code processing and analysis, especially in scenarios where a large number of code files or multiple language types need to be processed. In this way, multiple languages ​​can be processed simultaneously through multi-threaded concurrent processing, which improves the detection efficiency.

[0061] Table 2 Examples of detection languages ​​and detection file suffixes

[0062] Serial number Detect language Detect file suffix 1. java .java 2. Python .py 3. C .ch 4. cpp .cpp.cxx 5. vhdl .vhd.vhdl 6. Verilog .v,.sv,.vh

[0063] Then, according to the language rule mapping table, the source code file content is preprocessed, segmented and marked:

[0064] Set specific identifiers according to the programming language type, such as whether to perform escape processing, zero mark processing, etc.;

[0065] Read the comment rules, literal rules, and reserved word list in the language rule mapping table, and initialize the corresponding variables according to these rules.

[0066] Then read the source code content and convert it into a string.

[0067] After reading the stringified file content, it will be segmented and marked. In the segmentation process, the present invention traverses the entire file string character by character, and processes blank characters, indents, comments, literals, identifiers, numbers and symbols.

[0068] The specific process of word segmentation is as follows:

[0069] 1. Whitespace character processing: Whitespace characters are ignored, but the number of indents is calculated at the beginning of a line and processed accordingly based on whether spaces or tabs are used. If spaces or tabs are not used, they are not processed. If spaces or tabs are used, they are deleted and the number of spaces or tabs is calculated and recorded. At the same time, the line and column numbers before and after the deletion are used as markers;

[0070] 2. Indentation processing: record the start and end positions of the code block according to the indentation stack to ensure the correct identification of the code block structure. Delete the indentation and use the line and column numbers before and after the deletion as marks;

[0071] 3. Comment processing: By matching the start and end marks of comments, single-line and multi-line comments can be identified and skipped according to the comment rules in the language rule mapping table. For multi-line comments, nested comments must also be considered. Delete comments, and use the line and column numbers before and after the deletion as markers;

[0072] 4. Literal processing: According to the literal rules, it can identify and process literals such as strings and character constants, ensure the correct parsing of various literal formats, and use the row and column numbers of the literals as markers;

[0073] 5. Identifier and number processing: Match identifiers and numbers through regular expressions, and use the row and column numbers where the identifiers and numbers are located as tags. The present invention can identify variable names, function names, keywords, and various numerical constants.

[0074] 6. Symbol processing: Identify special symbols that do not belong to the above types, such as operators, delimiters, line breaks, and transfer characters, and use the row and column numbers of the symbols as marks.

[0075] During the entire word segmentation process, the marked content is: during the word segmentation process, the row and column position record and type processing of each word before and after processing.

[0076] After the word segmentation process is completed, the word segmentation results will be normalized, see Table 7 and Figure 3 As shown. Finally, a normalized source code file is generated for subsequent analysis and processing.

[0077] Table 3 Normalization after word segmentation

[0078] The result after word segmentation Normalize to the same type of value number Normalize to the same numeric type value String Normalize to the same string type value variable Normalize to the same variable type value, such as $ Keywords Do not normalize, keep the original value punctuation Normalize to the same identifier type value

[0079] The entire preprocessing process ensures accurate segmentation and tagging of the source code according to the mapping tables of different coding language rules. Through normalization, the source code of different coding languages ​​is unified into a common form, providing a reliable foundation for subsequent feature extraction.

[0080] Step three: Source code feature fingerprint extraction stage.

[0081] In this step, the source code file normalized in step 2 is segmented into code blocks according to the size of the feature extraction window and step size. Each code block is used as the minimum unit of feature extraction, and an MD5 hash value is generated for each code block, which is used as the feature fingerprint value of the code block.

[0082] The specific steps are as follows:

[0083] 1) Determine the window size and step size for feature extraction. The window size refers to a length or a certain number of code lines used to analyze the code, and the step size refers to the number of code lines that slide during analysis. The window and step size determine the granularity of feature extraction. Set the window size to N and the step size to M (N, M are both positive integers, and N>M). The smaller the values ​​of N and M, the smaller the window and step size, the smaller the code block for feature extraction, and the finer the granularity of extraction. The window and step size can be set according to actual usage requirements. In the specific implementation, by adjusting the values ​​of M and N, we can flexibly change the granularity of feature extraction to meet the needs of code feature analysis in different scenarios. The design of window size and step size not only enhances flexibility and applicability, but also takes into account the efficiency and accuracy of feature extraction.

[0084] 2) Segment the code blocks by window and step size. Segment the code blocks of the source code file content normalized in step 2 according to the window and step size in step 3 1). We use a sliding window method to segment the code blocks, and each window contains N lines of code. That is, according to the window size, the normalized source code file content is divided into a window of N lines as the minimum unit code block for feature extraction starting from the first line, and then slides with a step size of M lines. See Table 4.

[0085] Table 4 Code snippet extraction with window N and step size M

[0086] Code block number Fragment start line Fragment end line Block 1 1 N Block 2 1+M N+M Block 3 1+2M N+2M …… …… …… Block n 1+n*M N+n*M

[0087] 3) Generate MD5 hash value of code block. Generate MD5 hash value for each code block in step 3 (2), and the hash value is the characteristic value of the code block. This characteristic value can uniquely identify each code snippet. This unique characteristic value plays a key role in the global situation, ensuring the uniqueness of each code snippet, thereby maintaining integrity. It should be noted that if two code snippets have the same characteristics, their MD5 hash values ​​will be the same. This is determined by the MD5 hash algorithm, that is, mapping the same input to the same output. Therefore, when the contents of two code snippets are exactly the same, their characteristics will also be the same, and the same MD5 hash value will be generated. Because of this, the MD5 hash value can be used as a unique identifier of the code snippet to detect the same or similar code snippets. In this way, we can effectively perform code clone detection, similarity analysis and feature matching to ensure the accuracy and stability of code analysis.

[0088] 4) Output the code file with the hash value sequence as the characteristic fingerprint. In step 3), each code block in each window generates a unique MD5 hash value (characteristic value), and each normalized source code file generates a code file with the hash value sequence as the characteristic fingerprint. The hash value sequence of the file can be used as the characteristic fingerprint of the file. The output of the source code characteristic fingerprint extraction stage is all the code files in the project with the hash value sequence as the characteristic fingerprint.

[0089] Step 4: Information storage and output.

[0090] The code files with hash value sequences as feature fingerprints are serialized and stored, and additional information such as the number of lines in the file is counted. This information and the tags stored in step 2 will be stored in a standardized format, such as JSON format, for subsequent data analysis and visualization.

[0091] The technology of the present invention improves the efficiency and accuracy of code feature extraction, and can be applied to source code analysis, similarity detection, code clone detection, etc., to help developers and testers better understand and analyze source code during software intelligent analysis and processing, thereby improving the quality and efficiency of software development and testing.

[0092] The contents not described in detail in the specification of the present invention belong to the prior art known to the professional and technical personnel in this field.

Claims

1. An efficient code feature extraction method suitable for different programming languages, characterized by include: Load different programming language configuration files supported by this project from the specified project path, parse the corresponding programming language configuration files through the predefined language parsing rule mapper, and generate language rule mapping tables applicable to different programming languages; Identify the programming language type, create a multi-task queue and assign task threads according to the programming language type, pre-process the source code according to the language rule mapping table, perform word segmentation and tagging, normalize the word segmentation results and generate a normalized source code file; The normalized source code file is segmented into code blocks according to the given window and step size and a hash value is generated as a feature value to obtain a code file with a hash value sequence as a feature fingerprint; The code files and tag contents with hash value sequences as feature fingerprints are serialized, stored and output in a standardized format.

2. The efficient code feature extraction method applicable to different programming languages ​​according to claim 1, characterized in that: The programming language configuration file defines the syntax rules and feature extraction rules of the programming language; the current configuration files are mainly divided into two types: One is the formulation of rules about programming language grammar, which describes in detail the grammatical structure, keywords, operators and other language features of the language; The other is about the feature extraction rules of the language, which describes in detail how to identify and extract key features in the code.

3. The efficient code feature extraction method applicable to different programming languages ​​according to claim 1, characterized in that: When identifying a programming language type, the programming language type is determined by identifying a source code file suffix.

4. The efficient code feature extraction method applicable to different programming languages ​​according to claim 1, characterized in that: When creating a multi-task queue and allocating task threads by programming language type, one thread is allocated to each programming language to process the file matching tasks of the corresponding language.

5. The efficient code feature extraction method applicable to different programming languages ​​according to claim 1, characterized in that: The process of preprocessing the source code is as follows: Set specific identifiers according to the programming language type; Read the comment rules, literal rules, and reserved word list in the language rule mapping table, and initialize the relevant variables according to the corresponding rules; Read the source code content and convert it to a string.

6. The efficient code feature extraction method applicable to different programming languages ​​according to claim 5, characterized in that: In the process of word segmentation and tagging, the string generated by preprocessing is used to traverse the string generated by preprocessing character by character for word segmentation and tagging; The marked content is: during the word segmentation process, the row and column position and processing type of the processed content before and after each word segmentation process.

7. The efficient code feature extraction method applicable to different programming languages ​​according to claim 6, characterized in that: The specific processing content of word segmentation is as follows: Whitespace characters: Whitespace characters are ignored, but the whitespace characters at the beginning of the line are counted as indents; if spaces or tabs are used, they are deleted, and the number of spaces or tabs is counted and recorded; the line and column numbers before and after the deletion are used as markers; Indentation: Process the start and end positions of the code block according to the indentation stack and identify the code block structure; delete the indentation and use the line and column numbers before and after the deletion as markers; Comments: Identify and skip single-line and multi-line comments according to the comment rules in the language rule mapping table by matching the comment start and end markers; For multi-line comments, identify nested comments; delete comments, and use the line and column numbers before and after the deletion as markers; Literals: Identify and process literals according to literal rules, using the row and column numbers of the literals as markers; Identifiers and numbers: Identifiers and numbers are matched by regular expressions, and the row and column numbers of the identifiers and numbers are used as markers; Symbol: The present invention will directly use a single symbol as a mark, and directly use the row number and column number where the single symbol is located as the mark.

8. The efficient code feature extraction method applicable to different programming languages ​​according to claim 1, characterized in that: The specific normalization of the word segmentation results is as follows: Normalize the numbers after word segmentation to the same digital type value; Normalize the segmented strings into the same string type value; Normalize the variables after word segmentation to the same variable type value; The keywords after segmentation are not normalized and the original values ​​are kept; Normalize the punctuation after word segmentation to the same identifier type value.

9. The efficient code feature extraction method applicable to different programming languages ​​according to claim 1, characterized in that: The specific method of obtaining a code file with a hash value sequence as a characteristic fingerprint is as follows: S3-1 determines the window size and step size for feature extraction; S3-2 splits a code block from the normalized source code file according to the window; S3-3 generates a hash value of the code block as a characteristic value of the code block and records it; S3-4 moves the feature extraction window by the step length, returns to step S3-2 and continues, traverses all the codes, and obtains a code file with a hash value sequence as a feature fingerprint.

Images