A program language classification method, device and related equipment

Abstract

The application discloses a program language classification method, comprising: obtaining a code file to be classified; performing text filtering on the code file to be classified by using a word segmentation file to obtain target text; wherein the word segmentation file comprises entity word segmentation, and the entity word segmentation is a text symbol related to a program language type; and performing identification on the target text by using a preset classification model to obtain an identification result; the program language classification method can effectively improve the accuracy of a program language classification result. The application also discloses a program language classification device, a system and a computer readable storage medium, which have the above beneficial effects.

Description

Technical Field

[0001] This application relates to the field of natural language processing technology, and in particular to a method for classifying programming languages, as well as a programming language classification device, system and computer-readable storage medium. Background Technology

[0002] In related technologies, the classification of computer programming languages ​​generally employs methods such as file header matching and file extension differentiation, or classification methods based on artificial neural networks. However, the file header matching and file extension differentiation method cannot achieve accurate classification results when the file header and file extension are missing; furthermore, the classification accuracy drops significantly when files use special naming conventions or lack file extensions. The artificial neural network-based classification method only achieves high accuracy for code files that conform to coding standards; its accuracy is also very low for obfuscated code or code with non-standard syntax.

[0003] Therefore, improving the accuracy of programming language classification results is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0004] The purpose of this application is to provide a programming language classification method that can effectively improve the accuracy of programming language classification results; another purpose of this application is to provide a programming language classification device, system and computer-readable storage medium, all of which have the above-mentioned beneficial effects.

[0005] Firstly, this application provides a method for classifying programming languages, including: Obtain the code file to be categorized; The target text is obtained by filtering the code file to be classified using a word segmentation file; wherein the word segmentation file includes entity word segmentation, and the entity word segmentation is text symbols related to the programming language type. The target text is identified using a preset classification model to obtain the identification result.

[0006] Preferably, the preset classification model includes: a primary classification model and a secondary classification model; The step of using a preset classification model to identify the target text and obtain the identification result includes: The target text is identified using the primary classification model. If the target text matches the language category of a file of the same type, the target text is re-identified using the secondary classification model to obtain the identification result.

[0007] Preferably, the step of using a preset classification model to identify the target text and obtain the identification result includes: The target text is converted into text that the model can recognize using the Word2vec model. The preset classification model is used to identify the text that the model can recognize, and the recognition result is obtained.

[0008] Preferably, the preset classification model is the fastText network model.

[0009] Preferably, the step of using a word segmentation file to filter the code file to be classified to obtain the target text includes: The code file to be classified is divided into code blocks; The word segmentation file is used to filter the text of each code block to obtain the corresponding target text block; The step of using a preset classification model to identify the target text and obtain the identification result includes: The preset classification model is used to identify each target text block to obtain block identification results; The recognition results are obtained by calculating the recognition results of each block according to the preset weights.

[0010] Preferably, before using a preset classification model to identify the target text and obtain the identification result, the method further includes: The target text is identified using preset static rules; Determine whether a recognition result has been obtained; If not, then proceed with the step of using a preset classification model to identify the target text and obtain the identification result.

[0011] Secondly, this application provides a training method for a programming language classification model, including: Obtain each sample file; Obtain word segmentation files for filtering each of the sample files; wherein, the word segmentation files include entity word segments, and the entity word segments are text symbols related to the program language type; Based on the word segmentation file, each of the sample files is filtered to obtain the target sample text; Based on the target sample texts and their corresponding labels, a model is trained to obtain a preset classification model for programming language classification.

[0012] Preferably, obtaining the word segmentation file for filtering each of the sample files from each of the sample files includes: Non-entity word segmentation in the sample file is removed using preset rules to obtain the processed sample file; Entities that appear more than a preset number of times in the processed sample file are segmented into words, and the word segmentation file is generated.

[0013] Preferably, after training the model based on each of the target sample texts and the corresponding labels of each target sample text to obtain a preset classification model for programming language classification, the method further includes: When the classification accuracy is lower than a preset threshold, false alarm texts are identified, wherein the classification accuracy is: the classification accuracy of the identification results obtained by using the preset classification model to identify a preset number of target texts; The preset static rules are corrected based on the false alarm text to obtain the corrected static rules; wherein, the static rules are used to preprocess the target text corresponding to the code file to be classified.

[0014] Preferably, the training method for the programming language classification model further includes: When the classification accuracy is lower than the preset threshold, the preset classification model is optimized by modifying the model parameters to obtain an optimized classification model.

[0015] Thirdly, this application also discloses a programming language classification device, comprising: The file acquisition module is used to acquire the code files to be categorized. The text filtering module is used to filter the code file to be classified using a word segmentation file to obtain the target text; wherein, the word segmentation file includes entity word segmentation, and the entity word segmentation is text symbols related to the programming language type; The text recognition module is used to recognize the target text using a preset classification model and obtain the recognition result.

[0016] Fourthly, this application also discloses a training device for a programming language classification model, comprising: The sample acquisition module is used to acquire various sample files; The file generation module is used to obtain a word segmentation file for filtering each of the sample files from each of the sample files; wherein, the word segmentation file includes entity word segments, and the entity word segments are text symbols related to the program language type; The sample filtering module is used to filter each of the sample files based on the word segmentation file to obtain each target sample text; The model training module is used to train the model based on each of the target sample texts and the corresponding labels of each target sample text, so as to obtain a preset classification model for programming language classification.

[0017] Fifthly, this application also discloses a programming language classification system, including: Memory, used to store computer programs; A processor, configured to execute the computer program to implement the steps of any programming language classification method as described above and / or the steps of any programming language classification model training method as described above.

[0018] Sixthly, this application also discloses a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of any programming language classification method as described above and / or the steps of a training method for any programming language classification model as described above.

[0019] The programming language classification method provided in this application includes obtaining a code file to be classified; filtering the code file to be classified using a word segmentation file to obtain target text; wherein the word segmentation file includes entity word segmentation, and the entity word segmentation is text symbols related to the programming language type; and recognizing the target text using a preset classification model to obtain recognition results.

[0020] As can be seen, the programming language classification method provided in this application first uses a word segmentation file to filter text before classifying programming languages ​​using a classification model. This filters out text symbols that are not related to the code programming language and retains only the text symbols that are related to the code programming language. Since the word segmentation file includes various text symbols related to the code programming language, it can effectively reduce the impact of inaccurate code programming language classification results caused by the presence of confused code text segments in real-world scenarios, thereby improving the accuracy of code programming language classification in real-world scenarios.

[0021] The programming language classification device, system, and computer-readable storage medium provided in this application all have the aforementioned beneficial effects, which will not be elaborated further here. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the prior art and the embodiments of this application, the accompanying drawings used in the description of the prior art and the embodiments of this application will be briefly introduced below. Of course, the accompanying drawings described below with respect to the embodiments of this application are only a part of the embodiments in this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort, and such other drawings also fall within the protection scope of this application.

[0023] Figure 1 A flowchart illustrating a programming language classification method provided in this application; Figure 2A flowchart illustrating a training method for a programming language classification model provided in this application; Figure 3 A flowchart illustrating the training method for another programming language classification model provided in this application; Figure 4 A flowchart illustrating another programming language classification method provided in this application; Figure 5 A schematic diagram illustrating a programming language classification result based on the FastText model provided in this application; Figure 6 A schematic diagram of a programming language classification device provided in this application; Figure 7 A schematic diagram of the structure of a training device for a programming language classification model provided in this application; Figure 8 This is a schematic diagram of the structure of a programming language classification system provided in this application. Detailed Implementation

[0024] The core of this application is to provide a programming language classification method that can effectively improve the accuracy of programming language classification results; another core aspect of this application is to provide a programming language classification device, system, and computer-readable storage medium, which also have the aforementioned beneficial effects.

[0025] To provide a clearer and more complete description of the technical solutions in the embodiments of this application, the technical solutions in the embodiments of this application will be described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0026] This application provides a method for classifying programming languages.

[0027] Please refer to Figure 1 , Figure 1 This application provides a flowchart illustrating a programming language classification method, which may include: S101: Obtain the code file to be classified; This step aims to obtain the code file to be classified. This code file is the one whose programming language needs to be categorized. It can be directly input by the user through a client, or it can be a file collected and sent by other terminals or servers. This application does not limit the specific source of the code file to be classified. Furthermore, it should be noted that the language type of the code file to be classified does not affect the implementation of this technical solution. The code file mentioned in step S101 can specifically be a script code file.

[0028] S102: Use the word segmentation file to filter the code file to be classified to obtain the target text; wherein, the word segmentation file includes entity word segmentation, and the entity word segmentation is text symbols related to the programming language type; This step aims to filter the text of the code file to be classified using a word segmentation file. Specifically, after obtaining the code file to be classified, the word segmentation file can be used to filter its text. This file includes various entity segments, which refer to text symbols related to the programming language and can be extracted from pre-collected sample files. Furthermore, this word segmentation file can effectively filter out text symbols unrelated to the programming language in the code file to be classified, such as non-English symbols, consecutive special characters, consecutive spaces, and arrays. Thus, by using the word segmentation file to filter the text of the code file to be classified, the text symbols related to the programming language can be obtained, generating the target text. Based on this, classifying and recognizing only text symbols related to the programming language can effectively ensure the accuracy of the classification and recognition results.

[0029] S103: Use a preset classification model to identify the target text and obtain the identification result.

[0030] This step aims to achieve target text recognition to obtain the final recognition result. Specifically, after obtaining the target text, a preset classification model can be invoked to recognize it. That is, the target text is input into the preset classification model for processing to obtain the recognition result. The preset classification model is a pre-built machine learning model for classifying programming languages. It can be stored in a preset storage space, such as memory or storage device, so that it can be directly invoked when classifying and recognizing the target text.

[0031] As a preferred embodiment, the aforementioned preset classification model may include: a primary classification model and a secondary classification model; then, the above-mentioned recognition of target text using the preset classification model to obtain recognition results may include: recognizing the target text using the primary classification model; if the target text matches the language category corresponding to the same type of file, then recognizing the target text again using the secondary classification model to obtain recognition results.

[0032] Since there may be some highly similar texts in the actual code files, in order to ensure the accuracy of the classification results and to facilitate the collection of various different sample files, and to prevent overfitting caused by sample files consisting of only one type of highly similar text, a two-level classification model can be used to more accurately distinguish programming languages.

[0033] Specifically, during the model building process, a primary classification model and a secondary classification model can be built simultaneously. First, for sample files with high similarity (similarity exceeding a preset threshold), the two are merged into the same class of files, and then the primary classification model is trained with sample files from other classes. Further, the model is trained using only the aforementioned sample files with high similarity to obtain the secondary classification model, which is only used to classify and recognize two types of programming languages ​​with high similarity.

[0034] Based on this, in the process of classifying programming languages, after obtaining the target text, the primary classification model is first used for processing. If the target text matches the language category of the above-mentioned similar files, the secondary classification model can be used to further process it to obtain the final recognition result. If the target text does not match the language category of the above-mentioned similar files, the processing result can be directly used as the final recognition result. In this way, programming language classification and recognition are achieved.

[0035] Therefore, it can be seen that by constructing a multi-level classification model, programming language categories with high similarity can be identified, effectively ensuring the accuracy of the classification results.

[0036] As a preferred embodiment, the above-mentioned method of recognizing target text using a preset classification model to obtain recognition results may include: converting the target text into text that the model can recognize; and recognizing the model-recognizable text using a preset classification model to obtain recognition results.

[0037] Specifically, in the process of recognizing target text based on a preset classification model, the target text can first be converted into model-recognizable text, that is, text that the machine learning model (preset classification model) can recognize and process. Then, the preset classification model is used to process the model-recognizable text to obtain the final recognition result. In particular, converting the target text into model-recognizable text can be done by mapping each entity word in the target text into a vector of a set dimension (vectorization processing).

[0038] As a preferred embodiment, the above-described conversion of target text into model-recognizable text may include: using the Word2vec model (a related model used to generate word vectors) to convert the target text into model-recognizable text.

[0039] This preferred embodiment provides a method for entity segmentation vectorization, which uses the Word2vec model to convert target text into text that the model can recognize. The Word2vec model not only counts the impact of word frequency on the classification results but also considers the influence of relationships between words, effectively improving the accuracy of the classification results. Of course, the Word2vec model is only one implementation method provided in this preferred embodiment and is not the only one; other methods include the bag-of-words model, TF-IDF model, one-hot encoding, n-gram model, etc.

[0040] As a preferred embodiment, the aforementioned preset classification model can specifically be the fastText network model.

[0041] This preferred embodiment provides a specific type of preset classification model, namely the aforementioned fastText network model. Specifically, the fastText model is a library created by Facebook's AI Research lab for learning word embeddings and text classification. It uses neural networks for word embeddings, allowing the creation of unsupervised or supervised learning algorithms to obtain vector representations of words. Furthermore, the fastText model is a lightweight neural network model with small components, short training cycles, and low memory footprint.

[0042] As a preferred embodiment, the above-mentioned method of using a word segmentation file to filter text in a code file to be classified to obtain target text may include: dividing the code file to be classified to obtain code blocks; using the word segmentation file to filter text in each code block to obtain corresponding target text blocks; and then using a preset classification model to identify the target text to obtain identification results, which may include: using the preset classification model to identify each target text block to obtain block identification results; and calculating the identification results of each block according to preset weights to obtain identification results.

[0043] It is understandable that since some text in the actual code file may be obfuscated, it can easily lead to inaccurate model recognition results. Therefore, to solve this technical problem, the code can be divided into blocks for recognition, and then the final recognition result can be calculated by summarizing the results. Specifically, firstly, the code file to be classified is divided into multiple code blocks. The method of division does not affect the implementation of the technical solution, and this application does not limit it. Next, the text of each code block is filtered using a word segmentation file to obtain the corresponding target text block. Obviously, the process of filtering the text of each code block based on the word segmentation file is similar to the process of filtering the text of the code file to be classified based on the word segmentation file, and this application will not repeat it here. Next, the target text block is recognized using a preset classification model to obtain the recognition result of each target text block (i.e., the block recognition result mentioned above). Similarly, the process of recognizing the target text block based on the preset classification model is similar to the process of recognizing the target text based on the preset classification model mentioned above. Finally, the recognition results of each block are calculated according to preset weights to obtain the final recognition result.

[0044] The weight calculation process is as follows: Weighting coefficients can be assigned to each code block based on the actual situation. A larger weight value can be assigned to a code block that has a significant impact on the classification result, while a smaller weight value can be assigned. Clearly, the sum of the weight values ​​for each code block is 1. Therefore, after obtaining the block recognition results for each code block, the final classification result can be calculated by combining their respective weighting coefficients.

[0045] As a preferred embodiment, before the above-mentioned recognition of target text using a preset classification model to obtain recognition results, the method may further include: recognizing target text using preset static rules; determining whether recognition results are obtained; if not, then performing the step of recognizing target text using a preset classification model to obtain recognition results.

[0046] To effectively improve the efficiency of programming language classification, static rules can be pre-defined to preprocess the target text. This allows for direct classification of simple and easily categorized target text using these pre-defined static rules, eliminating the need for further classification using a pre-defined classification model. Specifically, after obtaining the target text from the word segmentation file, the pre-defined static rules can be invoked to identify the target text. If the corresponding recognition result can be obtained directly based on the pre-defined static rules, the programming language classification is complete, and there is no need to invoke the pre-defined classification model for further processing. Conversely, if the corresponding recognition result cannot be obtained based on the pre-defined static rules, the pre-defined classification model needs to be invoked again to identify the target text and obtain the final recognition result.

[0047] The specific type of the preset static rules is not unique. For example, it may include, but is not limited to, string matching rules, file header recognition rules, file extension recognition rules, etc. This application does not limit this.

[0048] As can be seen, the programming language classification method provided in this application first uses a word segmentation file to filter text before classifying programming languages ​​using a classification model. This filters out text symbols that are not related to the code programming language and retains only the text symbols that are related to the code programming language. Since the word segmentation file includes various text symbols related to the code programming language, it can effectively reduce the impact of inaccurate code programming language classification results caused by the presence of confused code text segments in real-world scenarios, thereby improving the accuracy of code programming language classification in real-world scenarios.

[0049] This application provides a method for training a programming language classification model.

[0050] Please refer to Figure 2 , Figure 2 A flowchart illustrating a training method for a programming language classification model provided in this application, the training method including: S201: Obtain each sample file; This step aims to obtain sample files, which are code files belonging to known programming language categories. It's conceivable that these sample files can also be used to construct subsequent word segmentation files. Furthermore, the more sample files available, the higher the accuracy of the pre-defined classification model, the more comprehensive the entity segmentation within the word segmentation files, and the more accurate the final classification result.

[0051] As a preferred embodiment, obtaining the sample files described above may include: obtaining the sample files from the network using web crawling technology.

[0052] This preferred embodiment provides a method for obtaining sample files, which is based on web crawling technology. It utilizes web crawling to scrape a large number of sample files from the entire internet. Of course, this implementation method is only one possible approach provided in this preferred embodiment and is not the only one.

[0053] S202: Obtain the word segmentation file used to filter each sample file from each sample file; wherein, the word segmentation file includes entity word segmentation, and entity word segmentation is text symbols related to the program language type; This step aims to generate a word segmentation file, which includes various entity segments. These entity segments refer to various text symbols related to the programming language, and can be extracted from the aforementioned sample files. It is conceivable that the word segmentation file can be used to filter the aforementioned sample files to obtain corresponding target sample files, thereby completing the construction of the preset classification model; it can also be used in the subsequent programming language classification process to filter the code files to be classified, obtaining the corresponding target text, and thus achieving programming language category recognition of the code files to be classified.

[0054] As a preferred embodiment, obtaining the word segmentation file for filtering each sample file from each sample file may include: removing non-entity word segments from the sample files using preset rules to obtain processed sample files; and filtering entity word segments that appear more than a preset number of times from the processed sample files to generate word segmentation files.

[0055] This preferred embodiment provides a process for generating a word segmentation file. Specifically, after obtaining the aforementioned sample files, non-entity word segments in the sample files can be removed using preset rules. These non-entity word segments are various text symbols unrelated to the programming language. The preset rules can be regular expressions, which can be used to match and exclude non-English characters, consecutive special characters, consecutive spaces, numbers, etc., while retaining special symbols related to the code programming language (such as underscores). Further, entity word segments with high frequency of occurrence (i.e., the number of occurrences exceeds a preset number) are selected from all entity word segments to generate a word segmentation file for subsequent filtering of code files to be classified. The value of the preset number of occurrences can be set by technicians according to the actual situation, and this application does not limit it.

[0056] S203: Filter each sample file based on the word segmentation file to obtain the target sample text; This step aims to filter the sample files and obtain the target sample file corresponding to each sample file. This target sample file is used to train the preset classification model. It is understood that the sample file filtering process based on word segmentation files is similar to the filtering process of the code files to be classified based on word segmentation files described above, and will not be elaborated upon here.

[0057] S204: Based on each target sample text and its corresponding label, train the model to obtain a preset classification model for programming language classification.

[0058] This step aims to train and generate a pre-defined classification model. As mentioned above, the sample files are code files with known programming language categories. These categories can be identified using labels. Therefore, after obtaining each target sample file, the model can be trained using its corresponding labels to obtain a pre-defined classification model for programming language classification. In the specific implementation, after obtaining each target sample file, it can also be converted into model-recognizable sample text. This model-recognizable sample text and its corresponding labels are then input into the initial machine learning model for training, resulting in the aforementioned pre-defined classification model.

[0059] As a preferred embodiment, after training the model based on each target sample text and its corresponding label to obtain a preset classification model for programming language classification, the method may further include: identifying false positive texts when the classification accuracy is lower than a preset threshold, wherein the classification accuracy is the classification accuracy obtained by recognizing a preset number of target texts using the preset classification model; and correcting the preset static rules based on the false positive texts to obtain corrected static rules; wherein the static rules are used for preprocessing the target texts corresponding to the code files to be classified. Alternatively, the static rules may not be corrected only when the classification accuracy is low; the classification accuracy may be disregarded, and the static rules may be corrected directly based on the actual false positives.

[0060] As mentioned above, to effectively improve the efficiency of programming language classification, static rules can be pre-defined to preprocess the target text. This allows for direct classification of simple and easily categorized target text using the pre-defined static rules, eliminating the need for further classification and recognition using a pre-defined classification model. Furthermore, to further improve the accuracy of programming language classification results, the pre-defined static rules can be modified based on the model processing results to obtain more accurate modified static rules.

[0061] In the specific implementation process, after constructing and obtaining the preset classification model, a preset number of target texts can be obtained, and the preset classification model can be used to identify each target text to obtain the corresponding classification results. Further, the classification accuracy of the preset classification model is calculated using all the classification results, that is, the ratio of the number of correctly classified results to the preset number. When the ratio is lower than the preset threshold, false alarm texts can be counted, and then the preset static rules can be corrected based on the false alarm texts. For example, a new static recognition rule corresponding to the false alarm text can be constructed and added to the preset static rules, thereby obtaining the corrected static rules.

[0062] The preset quantity mentioned above is not unique and can be set by technical personnel according to actual needs. It is understood that the larger the value, the more accurate the classification accuracy calculation result. In addition, the value of the preset threshold does not affect the implementation of this technical solution, and this application does not limit it.

[0063] As a preferred embodiment, the training method of the programming language classification model may further include: when the classification accuracy is lower than a preset threshold, optimizing the preset classification model by modifying the model parameters to obtain an optimized classification model.

[0064] As mentioned above, to further improve the accuracy of programming language classification results, the preset static rules can be modified based on the model processing results to obtain more accurate modified static rules. Simultaneously, the preset classification model can also be modified to improve model accuracy, thereby improving the accuracy of programming language classification results. Specifically, when the classification accuracy of the preset classification model is lower than the aforementioned preset threshold, it indicates that the classification accuracy of the preset classification model is low and needs optimization. In this case, the preset classification model can be optimized by modifying various model parameters to obtain an optimized classification model. It is understandable that the purpose of model optimization is to obtain a classification model with a classification accuracy reaching the preset threshold.

[0065] Based on the above embodiments, this application provides another method for classifying programming languages.

[0066] First, refer to Figure 3 , Figure 3 A flowchart illustrating a pre-defined classification model construction method provided in this application is shown below, and its specific implementation process is as follows: 1. Pre-defined classification model construction: (1) Submit code files (sample files): Specifically, standardized code files can be randomly collected from the entire network through methods such as web scraping. Furthermore, the content and quantity of the code files should be sufficient to meet the feature diversity requirements for subsequent model training, meaning that the code files can cover the programming language classes that need to be classified.

[0067] (2) Constructing the word segmentation file: Specifically, entities in the code text can be matched using preset rules. Entities that appear more than a first preset threshold are selected as words and recorded in a word segmentation file. The preset rules use regular expressions to match and exclude non-English characters, consecutive special characters, consecutive spaces, numbers, etc., while retaining special characters related to the code programming language (such as underscores), generating a word segmentation file for subsequent filtering of the code text file.

[0068] (3) Constructing a classification model: Specifically, the word segmentation file in 1-(2) can be used to filter the sample file in 1-(1) to obtain the corresponding target sample file; further, the filtered target sample file is represented by text, that is, it is converted into a representation that can be recognized and processed by the subsequent machine learning model. The text representation process is implemented using the word2vec model.

[0069] In practical implementation, the first step is to build and train an artificial neural network to perform word vectorization, that is, to map each entity word in the word segmentation file into a vector of a set dimension. Then, the vectors of all entity words in a code file are stacked and averaged. Finally, the calculated value is applied to an activation function and matched with a known code programming language type to complete the training of the classification model, thus obtaining the classification model. The classification model can be a lightweight neural network model with fewer components, such as FastText.

[0070] Furthermore, when the classification model needs to add new programming language classes that can be classified, in step 1-(1) of submitting code files, code files for the required programming language classes can be randomly collected from the entire network through methods such as web scraping. The content and quantity of these code files should meet the feature diversity requirements for subsequent model training. Further, continue with the following steps 1-(2) and 1-(3) to finally train a classification model that can recognize and classify new programming language classes, thus updating the classification model. In both the preprocessing stage for adding new programming language classes and the model training stage, the accuracy of the updated model can be ensured by adjusting relevant parameters, thereby guaranteeing the accuracy of the classification results.

[0071] Further, refer to Figure 4 , Figure 4 A flowchart illustrating another programming language classification method provided in this application is shown below, and its specific implementation process is as follows: 2. Classification of programming languages: (1) Submit the code files that need to be categorized: For the submitted code file set, preprocessing and classification operations can be performed on single files or multiple files in batches. The following describes the programming language classification method provided in this application using a single file as an example. This code file contains code content of random types within a preset class.

[0072] (2) Code file preprocessing based on word segmentation files: The word segmentation file constructed in 1-(2) is used to filter the code text content of the code file that needs to be classified, so as to obtain the text content that can be used for subsequent classification.

[0073] (3) Classification of programming languages ​​based on preset static rules: The text content obtained in step 2-(2) is classified into programming languages ​​using preset static rules. If the corresponding recognition result can be obtained, the programming language classification process ends and no further steps are needed. If the corresponding recognition result cannot be obtained, the process can proceed to step 2-(4) to use the classification model for re-recognition. The preset static rules include, but are not limited to, static rules such as string matching, file header recognition, and file extension recognition.

[0074] (4) Classification of programming languages ​​based on classification models: The classification model constructed in 1-(3) is used to represent and classify the filtered text content in 2-(2) to obtain the final classification result.

[0075] This allows for the classification of programming languages.

[0076] Furthermore, a classification model can be used to identify and process batches of files, and the classification accuracy can be calculated. If the classification accuracy significantly falls below a preset threshold, the code files with false positives / classification errors can be examined. New static rules can be extracted by manually observing the files and then added to the preset static rules to correct them. Of course, when the classification accuracy does not reach the preset threshold, the model accuracy can also be improved by adjusting relevant model parameters to obtain a classification model that reaches the preset threshold.

[0077] Finally, based on the programming language classification method provided in the embodiments of this application, for a dataset randomly sampled from the entire network, the accuracy rate is as high as 90% or more after testing. Figure 5 As shown, Figure 5 This is a schematic diagram of the programming language classification results based on the FastText model provided in this application; and, for some types containing obfuscated code, due to the continuous optimization of the preprocessing process in the subsequent recognition process, the recognition accuracy can be as high as 80% or more.

[0078] As can be seen, the programming language classification method provided in this application first uses a word segmentation file to filter text before classifying programming languages ​​using a classification model. This filters out text symbols that are not related to the programming language and retains only the text symbols that are related to the programming language. Since the word segmentation file includes various text symbols related to the programming language, it can effectively reduce the impact of inaccurate classification results of programming languages ​​due to the presence of confused code text segments in real-world scenarios, thereby improving the accuracy of programming language classification in real-world scenarios.

[0079] To address the aforementioned technical problems, this application also provides a programming language classification device, please refer to... Figure 6 , Figure 6 This application provides a schematic diagram of the structure of a programming language classification device, which may include: File acquisition module 1 is used to acquire code files to be classified. Text filtering module 2 is used to filter the text of the code file to be classified using the word segmentation file to obtain the target text; wherein, the word segmentation file includes entity word segmentation, which consists of text symbols related to the programming language type; Text recognition module 3 is used to recognize target text using a preset classification model and obtain recognition results.

[0080] As can be seen, the programming language classification device provided in this application first uses a word segmentation file to filter text before classifying programming languages ​​using a classification model, so as to filter out text symbols that are not related to the code programming language and retain only the text symbols related to the code programming language. Since the word segmentation file includes various text symbols related to the code programming language, it can effectively reduce the impact of inaccurate code programming language classification results caused by the presence of confused code text segments in actual scenarios, and improve the accuracy of code programming language classification in actual scenarios.

[0081] As a preferred embodiment, the aforementioned preset classification model may include a primary classification model and a secondary classification model; then the aforementioned text recognition module 3 may be specifically used to recognize the target text using the primary classification model; if the target text matches the language category of the same type of file, the target text is recognized again using the secondary classification model to obtain the recognition result.

[0082] In a preferred embodiment, the text recognition module 3 described above may include: The text conversion unit is used to convert target text into text that the Word2vec model can recognize, using the Word2vec model. The text recognition unit is used to recognize text that the model can recognize using a preset classification model, and obtain the recognition result.

[0083] As a preferred embodiment, the aforementioned preset classification model can specifically be the fastText network model.

[0084] In a preferred embodiment, the text filtering module 2 can be specifically used to divide the code file to be classified to obtain each code block; to filter each code block using the word segmentation file to obtain the corresponding target text block; then the text recognition module 3 can be specifically used to recognize each target text block using a preset classification model to obtain the block recognition result; and to calculate the recognition result of each block according to the preset weight.

[0085] In a preferred embodiment, the programming language classification device may further include a preprocessing unit, used to identify the target text using preset static rules before the target text is identified using the preset classification model to obtain the identification result; determine whether an identification result is obtained; if not, then execute the step of identifying the target text using the preset classification model to obtain the identification result.

[0086] For a description of the device provided in this application, please refer to the above method embodiments; further details will not be provided here.

[0087] To address the aforementioned technical problems, this application also provides a training device for a programming language classification model. Please refer to [reference needed]. Figure 7 , Figure 7 A schematic diagram of a training device for a programming language classification model provided in this application. The training device for the programming language classification model may include: Sample acquisition module 4 is used to acquire each sample file; File construction module 5 is used to obtain word segmentation files for filtering each sample file from each sample file; the word segmentation files include entity word segmentation, which consists of text symbols related to the program language type; Sample filtering module 6 is used to filter each sample file based on the word segmentation file to obtain the target sample text; Model training module 7 is used to train the model based on each target sample text and the corresponding label of each target sample text, so as to obtain a preset classification model for programming language classification.

[0088] In a preferred embodiment, the file construction module 5 can be specifically used to remove non-entity segmented words from the sample file using preset rules to obtain a processed sample file; and to filter entity segmented words that appear more than a preset number of times from the processed sample file to generate a segmented word file.

[0089] In a preferred embodiment, the training device for the programming language classification model may further include a static rule correction module. This module is used to determine false positive texts when the classification accuracy is lower than a preset threshold after the model training is performed based on each target sample text and its corresponding label. The classification accuracy is calculated by using the preset classification model to identify a preset number of target texts, resulting in a classification accuracy rate. The module then corrects the preset static rules based on the false positive texts to obtain corrected static rules. These static rules are used to preprocess the target text corresponding to the code file to be classified.

[0090] In a preferred embodiment, the training device for the programming language classification model may further include a model optimization module, which is used to optimize the preset classification model by modifying the model parameters when the classification accuracy is lower than a preset threshold, so as to obtain an optimized classification model.

[0091] For a description of the device provided in this application, please refer to the above method embodiments; further details will not be provided here.

[0092] To address the aforementioned technical problems, this application also provides a programming language classification system, please refer to... Figure 8 , Figure 8 This application provides a schematic diagram of the structure of a programming language classification system. The programming language classification device may include: Memory 10 is used to store computer programs; The processor 20 is configured to execute computer programs and implement steps according to any of the programming language classification methods described above.

[0093] For a description of the system provided in this application, please refer to the above method embodiments; further details will not be provided here.

[0094] To address the aforementioned problems, this application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, can implement the steps of any of the programming language classification methods described above.

[0095] The computer-readable storage medium may include various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0096] For a description of the computer-readable storage medium provided in this application, please refer to the above method embodiments; further details will not be repeated here.

[0097] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0098] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0099] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0100] The technical solutions provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are only for the purpose of helping to understand the methods and core ideas of this application. It should be noted that those skilled in the art can make several improvements and modifications to this application without departing from the principles of this application, and these improvements and modifications also fall within the protection scope of this application.

Claims

1. A method for classifying programming languages, characterized in that, include: Obtain the code file to be categorized; The target text is obtained by filtering the code file to be classified using a word segmentation file; wherein the word segmentation file includes entity word segmentation, and the entity word segmentation is text symbols related to the programming language type. The target text is identified using a preset classification model to obtain the identification result; The process of generating the word segmentation file includes: obtaining each sample file; removing non-entity word segments from the sample files using preset rules to obtain processed sample files; filtering entity word segments that appear more than a preset number of times from the processed sample files to generate the word segmentation file; the preset rules are regular expressions. The preset classification model includes a primary classification model and a secondary classification model. The primary classification model is trained based on sample file sets of different categories, and the similarity between sample files in each sample file set exceeds a preset threshold, while the similarity between sample files in different sample file sets does not exceed the preset threshold. The secondary classification model is trained based on sample files of the same category, and the similarity between sample files exceeds the preset threshold. The step of using a preset classification model to identify the target text and obtain the identification result includes: using the primary classification model to identify the target text; if the target text matches the language category of the same type of file, then using the secondary classification model to identify the target text again and obtain the identification result.

2. The programming language classification method according to claim 1, characterized in that, The step of using a preset classification model to identify the target text and obtain the identification result includes: The target text is converted into text that the model can recognize using the Word2vec model. The preset classification model is used to identify the text that the model can recognize, and the recognition result is obtained.

3. The programming language classification method according to claim 1, characterized in that, The preset classification model is specifically the fastText network model.

4. The programming language classification method according to claim 1, characterized in that, The step of using a word segmentation file to filter the code file to be classified to obtain the target text includes: The code file to be classified is divided into code blocks; The word segmentation file is used to filter the text of each code block to obtain the corresponding target text block; The step of using a preset classification model to identify the target text and obtain the identification result includes: The preset classification model is used to identify each target text block to obtain block identification results; The recognition results are obtained by calculating the recognition results of each block according to the preset weights.

5. The programming language classification method according to any one of claims 1 to 4, characterized in that, Before using a preset classification model to identify the target text and obtain the identification result, the method further includes: The target text is identified using preset static rules; Determine whether a recognition result has been obtained; If not, then proceed with the step of using a preset classification model to identify the target text and obtain the identification result.

6. A training method for a programming language classification model, characterized in that, include: Obtain each sample file; Obtain word segmentation files for filtering each of the sample files; wherein, the word segmentation files include entity word segments, and the entity word segments are text symbols related to the program language type; Based on the word segmentation file, each of the sample files is filtered to obtain the target sample text; Based on the target sample texts and the corresponding labels of each target sample text, the model is trained to obtain a preset classification model for programming language classification. The step of obtaining the word segmentation file for filtering each of the sample files includes: removing non-entity word segments from the sample files using preset rules to obtain a processed sample file; filtering entity word segments that appear more than a preset number of times from the processed sample file to generate the word segmentation file; the preset rules are regular expressions. The preset classification model includes a primary classification model and a secondary classification model. The primary classification model is trained based on sample file sets of different categories, and the similarity between sample files in each sample file set exceeds a preset threshold, while the similarity between sample files in different sample file sets does not exceed the preset threshold. The secondary classification model is trained based on sample files of the same category, and the similarity between sample files exceeds the preset threshold. The preset classification model is used to identify the target text using the primary classification model; if the target text matches the language category of the same type of file, the secondary classification model is used to identify the target text again to obtain the identification result.

7. The training method according to claim 6, characterized in that, After training the model based on each target sample text and its corresponding label to obtain a preset classification model for programming language classification, the process further includes: When the classification accuracy is lower than a preset accuracy threshold, false alarm texts are identified, wherein the classification accuracy is: the classification accuracy of the identification results obtained by using the preset classification model to identify a preset number of target texts; The preset static rules are corrected based on the false alarm text to obtain the corrected static rules; wherein, the static rules are used to preprocess the target text corresponding to the code file to be classified.

8. The training method according to claim 7, characterized in that, Also includes: When the classification accuracy is lower than the preset accuracy threshold, the preset classification model is optimized by modifying the model parameters to obtain an optimized classification model.

9. A programming language classification device, characterized in that, include: The file acquisition module is used to acquire the code files to be categorized. The text filtering module is used to filter the code file to be classified using a word segmentation file to obtain the target text; wherein, the word segmentation file includes entity word segmentation, and the entity word segmentation is text symbols related to the programming language type; The text recognition module is used to recognize the target text using a preset classification model and obtain the recognition result; The word segmentation file generation module is used to obtain various sample files; remove non-entity word segments from the sample files using preset rules to obtain processed sample files; and filter out entity word segments that appear more than a preset number of times from the processed sample files to generate the word segmentation file; the preset rules are regular expressions. The preset classification model includes a primary classification model and a secondary classification model. The primary classification model is trained based on sample file sets of different categories, and the similarity between sample files in each sample file set exceeds a preset threshold, while the similarity between sample files in different sample file sets does not exceed the preset threshold. The secondary classification model is trained based on sample files of the same category, and the similarity between sample files exceeds the preset threshold. The text recognition module is specifically used to recognize the target text using the primary classification model; if the target text matches the language category of the same type of file, the secondary classification model is used to recognize the target text again to obtain the recognition result.

10. A training device for a programming language classification model, characterized in that, include: The sample acquisition module is used to acquire various sample files; The file generation module is used to obtain a word segmentation file for filtering each of the sample files from each of the sample files; wherein, the word segmentation file includes entity word segments, and the entity word segments are text symbols related to the program language type; The sample filtering module is used to filter each of the sample files based on the word segmentation file to obtain each target sample text; The model training module is used to train the model based on each of the target sample texts and the corresponding labels of each of the target sample texts, so as to obtain a preset classification model for programming language classification. Specifically, the file generation module is used to remove non-entity segmented words from the sample file using preset rules to obtain a processed sample file; and to filter out entity segmented words that appear more than a preset number of times from the processed sample file to generate the segmented word file; the preset rules are regular expressions. The preset classification model includes a primary classification model and a secondary classification model. The primary classification model is trained based on sample file sets of different categories, and the similarity between sample files in each sample file set exceeds a preset threshold, while the similarity between sample files in different sample file sets does not exceed the preset threshold. The secondary classification model is trained based on sample files of the same category, and the similarity between sample files exceeds the preset threshold. The preset classification model is used to identify the target text using the primary classification model; if the target text matches the language category of the same type of file, the secondary classification model is used to identify the target text again to obtain the identification result.

11. A programming language classification system, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the programming language classification method as described in any one of claims 1 to 5 and / or the steps of the training method for the programming language classification model as described in any one of claims 6 to 8 when executing the computer program.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the programming language classification method as described in any one of claims 1 to 5 and / or the steps of the training method for the programming language classification model as described in any one of claims 6 to 8.

Images