Source code identification method and device fusing cross-domain negative samples, equipment and medium
By integrating cross-domain negative sample methods and deep learning model training, the problem of misjudgment of non-code text in existing technologies has been solved, achieving efficient source code recognition and improving recognition accuracy and efficiency.
Patent Information
- Application Number
- CN202511422677.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-09-30
- Publication Date
- 2026-01-13
AI Technical Summary
Existing technologies lack an effective mechanism for distinguishing non-code text, which makes it easy for non-code text to be misidentified as source code. In particular, when faced with diverse programming languages and complex text environments, the accuracy of source code recognition is low.
We employ a method that integrates cross-domain negative samples. By collecting source code and non-source code samples from multiple programming languages, we train the VDCNN deep learning model and combine it with data preprocessing strategies to identify source code and non-source code text.
It significantly improves the accuracy of source code recognition, effectively avoids non-code text being misidentified as source code, enhances the accuracy and efficiency of recognition, and adapts to diverse text environments.
Smart Images
Figure CN121327140A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of computer technology, specifically to a method, apparatus, device, and medium for source code identification that integrates cross-domain negative samples. Background Technology
[0002] In the field of computer technology, with the continuous expansion of software development scale and the widespread application of open source code, the effective identification of source code through text classification technology has also developed.
[0003] Source code recognition technology not only facilitates code management and review but also plays a role in information security. However, current technologies lack effective mechanisms for distinguishing non-code text, such as Chinese text and English conversational text, easily misidentifying non-code text as source code. In practical applications, text types are complex and diverse; besides source code from various programming languages, there is a large amount of non-code text. Therefore, researching methods to distinguish between source code and non-code text and accurately identify source code types is of significant value. Summary of the Invention
[0004] The purpose of this application is to provide a source code recognition method, apparatus, electronic device, and storage medium that integrates cross-domain negative samples, which can improve the accuracy of source code recognition.
[0005] In a first aspect, embodiments of this application provide a source code identification method that integrates cross-domain negative samples, including:
[0006] The text to be tested is preprocessed to obtain the target text;
[0007] The target text is input into a pre-trained source code recognition model to obtain the recognition result output by the source code recognition model, and the recognition result indicates the category of the text to be tested; wherein, the source code recognition model is obtained by training the VDCNN model based on a sample set, and the sample set includes source code samples and non-source code samples.
[0008] In some embodiments of this application, the source code recognition model is pre-trained in the following manner:
[0009] Source code files in various programming languages were collected from the GitHub open-source platform and organized according to different programming language categories to obtain source code samples;
[0010] Collect English news articles, novel texts, and Q&A records, and label them as dialog; collect English gibberish and Latin texts, and label them as other; the texts in the dialog and other categories constitute non-source code samples; the source code samples and the non-source code samples constitute a sample set;
[0011] The sample set is preprocessed and then divided into a training set and a test set.
[0012] The VDCNN model is trained based on the training set and test set, and the source code recognition model is obtained after the preset training cutoff condition is met.
[0013] In some embodiments of this application, the preprocessing operations include at least one of the following: uniformly encoding the text; removing comments from various types of code using regular expressions; eliminating completely duplicate code files by adopting a deduplication strategy based on file hash values; and excluding cross-language mixed files through syntax analysis.
[0014] In some embodiments of this application, the step of preprocessing the text to be tested to obtain the target text includes:
[0015] The text to be tested is read using UTF-8 encoding to obtain the corresponding string;
[0016] The target text is determined based on the character length of the string.
[0017] In some embodiments of this application, determining the recognition strategy based on the character length of the string includes:
[0018] If the character length of the string is less than 2048 characters, then the full text of the string is determined as the target text;
[0019] If the character length of the string is greater than or equal to 2048 characters and less than or equal to 4096 characters, then the first 1024 characters and the last 1024 characters of the string are determined as the target text;
[0020] If the length of the string is greater than 4096 characters, then the first 1024 characters and the last 1024 characters of the string, as well as three randomly sampled text segments from the string, are determined as the target text.
[0021] In some embodiments of this application, before determining the target text based on the character length of the string, the method further includes:
[0022] Determine the proportion of wide characters and symbols in the string. If the proportion of wide characters or symbols exceeds a preset value, the text to be tested is determined to be non-source code text and returned to the "other" category.
[0023] The subsequent steps will not be performed.
[0024] Secondly, embodiments of this application provide a source code recognition device that integrates cross-domain negative samples, comprising:
[0025] The preprocessing module is used to preprocess the text to be tested to obtain the target text;
[0026] The source code recognition module is used to input the target text into a pre-trained source code recognition model and obtain the recognition result output by the source code recognition model, wherein the recognition result indicates the category of the text to be tested;
[0027] The source code recognition model is obtained by training the VDCNN model based on a sample set, which includes source code samples and non-source code samples.
[0028] In some embodiments of this application, the apparatus further includes: a model training module for pre-training the source code recognition model in the following manner:
[0029] Source code files in various programming languages were collected from the GitHub open-source platform and organized according to different programming language categories to obtain source code samples;
[0030] Collect English news articles, novel texts, and Q&A records, and label them as dialog; collect English gibberish and Latin texts, and label them as other; the texts in the dialog and other categories constitute non-source code samples; the source code samples and the non-source code samples constitute a sample set;
[0031] The sample set is preprocessed and then divided into a training set and a test set.
[0032] The VDCNN model is trained based on the training set and test set, and the source code recognition model is obtained after the preset training cutoff condition is met.
[0033] In some embodiments of this application, the preprocessing operations include at least one of the following: uniformly encoding the text; removing comments from various types of code using regular expressions; eliminating completely duplicate code files by adopting a deduplication strategy based on file hash values; and excluding cross-language mixed files through syntax analysis.
[0034] In some embodiments of this application, the preprocessing module is specifically used for:
[0035] The text to be tested is read using UTF-8 encoding to obtain the corresponding string;
[0036] The target text is determined based on the character length of the string.
[0037] In some embodiments of this application, the preprocessing module is specifically used for:
[0038] If the character length of the string is less than 2048 characters, then the full text of the string is determined as the target text;
[0039] If the character length of the string is greater than or equal to 2048 characters and less than or equal to 4096 characters, then the first 1024 characters and the last 1024 characters of the string are determined as the target text;
[0040] If the length of the string is greater than 4096 characters, then the first 1024 characters and the last 1024 characters of the string, as well as three randomly sampled text segments from the string, are determined as the target text.
[0041] In some embodiments of this application, the preprocessing module is further configured to determine the proportion of wide characters and symbols in the string before determining the target text based on the character length of the string; if the proportion of wide characters or symbols exceeds a preset ratio, the text to be tested is determined to be non-source code text and returned to the other category.
[0042] The subsequent steps will not be performed.
[0043] Thirdly, this application provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method as described in the first aspect.
[0044] Fourthly, this application provides a computer-readable storage medium having computer-readable instructions stored thereon, which can be executed by a processor to implement the method described in the first aspect.
[0045] Compared to existing technologies, the source code recognition method fusion method for cross-domain negative samples provided in this application preprocesses the test text to obtain the target text; the target text is then input into a pre-trained source code recognition model to obtain the recognition result output by the source code recognition model, wherein the recognition result indicates the category of the test text. The source code recognition model of this application is based on a sample set...
[0046] The sample set, trained using the VDCNN model, includes both source code and non-source code samples. Therefore, this application addresses the problem of insufficient ability to distinguish non-code text in existing technologies. When training the source code recognition model, it introduces texts such as English news, Yahoo! Answers records, and novel excerpts as cross-domain negative samples at the source code sample level across multiple programming language categories. This effectively avoids detecting non-code text as source code, thus demonstrating higher recognition accuracy in real-world file processing scenarios. Attached Figure Description
[0047] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:
[0048] Figure 1 A flowchart of a source code identification method for fusing cross-domain negative samples provided in an embodiment of this application is shown;
[0049] Figure 2 A flowchart of a source code recognition model training method provided in an embodiment of this application is shown;
[0050] Figure 3 This paper presents an overall flowchart of a source code identification method that integrates cross-domain negative samples, as provided in an embodiment of this application.
[0051] Figure 4 This illustrates the entire process of sample collection and preprocessing provided in the embodiments of this application;
[0052] Figure 5 This illustration shows a schematic diagram of the structure of a source code recognition device that integrates cross-domain negative samples, as provided in an embodiment of this application.
[0053] Figure 6 A schematic diagram of an electronic device provided in an embodiment of this application is shown. Detailed Implementation
[0054] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.
[0055] It should be noted that, unless otherwise stated, the technical or scientific terms used in this application shall have the ordinary meaning as understood by one of ordinary skill in the art to which this application pertains.
[0056] Furthermore, the terms "first" and "second," etc., are used to distinguish different objects, not to describe a specific order. Additionally, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to those processes, methods, products, or apparatuses.
[0057] The existing methods for distinguishing between source code and non-code text are as follows:
[0058] (1) Traditional methods typically utilize techniques such as lexical analysis and keyword analysis to extract specific syntactic structures and keywords from source code for classification, such as the TF-IDF algorithm. TF-IDF (Term Frequency-Inverse Document Frequency) is a classic text feature extraction algorithm in the fields of information retrieval and text mining. Traditional methods lack flexibility, rely on manual feature extraction, have weaker generalization ability than deep learning, are difficult to adapt to diverse programming languages and complex text environments, and have low source code recognition accuracy.
[0059] (2) Deep learning-based methods automatically learn the feature representations of source code text by constructing convolutional neural network models, thereby achieving source code classification, such as the Guesslang open-source language recognition tool. Although deep learning-based algorithms can automatically learn source code features, they lack the ability to effectively distinguish non-code text, and are prone to misjudgment, especially when faced with English non-code text.
[0060] (3) Rule-based methods detect languages by considering naming conventions, extracting file extensions, and analyzing file headers and comments to identify code categories, such as the Linguist program developed by Github.
[0061] Rule-based methods match pre-defined language keywords, but they cannot handle variations such as abbreviations and annotations, resulting in a high false positive rate and low efficiency in recognizing large texts.
[0062] This application addresses the shortcomings of existing technologies in distinguishing non-code text by proposing a source code recognition method that integrates cross-domain negative samples. At the source code sample level of multiple programming language categories (such as Assembly, C, C++, Python, etc.), texts including English news, Yahoo! Answers records, and novel excerpts are introduced as cross-domain negative samples. The model is trained using the PyTorch deep learning framework, and combined with data preprocessing strategies, effectively preventing non-code text from being detected as source code, thus demonstrating higher recognition accuracy in real-world file processing scenarios.
[0063] Please refer to Figure 1 , Figure 1 This is a flowchart of a source code identification method for fusing cross-domain negative samples provided in an embodiment of this application, including the following steps S101 to S102:
[0064] S101. Perform data preprocessing on the text to be tested to obtain the target text.
[0065] Specifically, step S101 can be implemented as follows: read the text to be tested using UTF-8 encoding to obtain the corresponding string; determine the target text based on the character length of the string.
[0066] Specifically, the steps described above for determining the target text based on the character length of the string can be implemented as follows:
[0067] If the character length of the string is less than 2048 characters, then the full text of the string is determined as the target text;
[0068] If the character length of the string is greater than or equal to 2048 characters and less than or equal to 4096 characters, then the first 1024 characters and the last 1024 characters of the string are determined as the target text;
[0069] If the length of the string is greater than 4096 characters, then the first 1024 characters and the last 1024 characters of the string, as well as three randomly sampled text segments from the string, are determined as the target text.
[0070] In some implementations, before the step of determining the target text based on the character length of the string, the method may further include: determining the proportion of wide characters and symbols in the string; if the proportion of wide characters or symbols exceeds a preset ratio, the text to be tested is determined to be non-source code text and returned to the other category; and subsequent steps are not performed.
[0071] The preset ratio can be set to 20%, 50%, 80%, etc., and this application does not limit it.
[0072] S102. Input the target text into a pre-trained source code recognition model to obtain the recognition result output by the source code recognition model, wherein the recognition result indicates the category of the text to be tested;
[0073] The source code recognition model is obtained by training the VDCNN model based on a sample set, which includes source code samples and non-source code samples.
[0074] like Figure 2 As shown, the source code recognition model can be pre-trained in the following manner:
[0075] S201. Collect source code files of various programming languages from the Github open-source code platform, and organize them according to different programming language categories to obtain source code samples;
[0076] S202. Collect English news articles, novel texts, and Q&A records, and label them as dialog category; collect English gibberish and Latin texts, and label them as other category; the texts of the dialog category and the other category constitute non-source code samples; the source code samples and the non-source code samples constitute a sample set;
[0077] S203. Perform preprocessing operations on the sample set, and divide the preprocessed sample set into a training set and a test set;
[0078] The preprocessing operations include at least one of the following: uniformly encoding the text; removing comments from various types of code using regular expressions; eliminating completely duplicate code files by adopting a deduplication strategy based on file hash values; and detecting and excluding cross-language mixed files through syntax analysis.
[0079] S204. The VDCNN model is trained based on the training set and the test set, and the source code recognition model is obtained after the preset training cutoff condition is met.
[0080] For ease of understanding, this application provides the following: Figure 3 The flowchart shown is the overall process of the source code identification method that integrates cross-domain negative samples.
[0081] I. The sample collection and preprocessing process is as follows:
[0082] like Figure 4 The diagram shows the entire process of sample collection and preprocessing.
[0083] 1. Source code sample collection: Source code files of more than 20 programming languages such as Assembly, C, C++, and Python were collected from the Github open source code platform and organized according to different language categories.
[0084] 2. Non-Source Code Sample Collection: English news articles, novel texts, and question-and-answer records are collected and labeled as "dialog"; English gibberish and Latin text are collected and labeled as "other". By selecting texts that differ significantly from the source code in semantics and structure as negative samples, the model's ability to distinguish non-code text is enhanced. Specifically,
[0085] 1) Semantic differences: Source code samples express computer-executable logical operations and data structures (such as variable operations, flow control, memory management, etc.), while non-code text is usually natural language text, which has no direct operational meaning for computers.
[0086] Positive sample example (C++):
[0087] int main(){vector<int>arr={1,2,3};for(auto x:arr)cout< <x*2;}。
[0088] Negative sample example (news / dialog):
[0089] "The stock market surged 3.2% yesterday amid renewed investor confidence" (economic phenomenon description only).
[0090] "&Xk8#j@Zq*2%P^G$!Ld9s?(garbled text)
[0091] 2) Structural differences: Source code samples strictly follow the syntax rules of a specific programming language (such as keywords, operators, delimiters, and indentation), exhibiting a well-defined hierarchical structure. Non-code text structures are free and loose, following natural language rules or having no rules at all.
[0092] Positive sample example (Python):
[0093] deffactorial(n):
[0094] if n <= 1:
[0095] return 1
[0096] return n*factorial(n-1).
[0097] Negative sample examples (novel / dialog):
[0098] "I can't go on," she whispered, trembling fingers brushing away tears. (Natural Language Structure)
[0099] Preprocessing steps: First, the text is uniformly encoded to ensure sample consistency; regular expressions are used to remove comments from various types of code; a deduplication strategy based on file hash values is employed to eliminate completely duplicate code files. Simultaneously, syntax analysis is used to detect and exclude cross-language mixed files (such as C header files mistakenly labeled as C++). Finally, the data is divided into training and test sets at a 10:1 ratio.
[0100] II. The model construction and training process is as follows:
[0101] 1. Constructing a Deep Learning Model: The VDCNN model was chosen as the core architecture, and the PyTorch framework was used to construct the VDCNN model. This model efficiently captures local and global features of text through multi-layer convolutional operations and stacks multiple residual structures, thus effectively handling the grammatical and structural patterns of source code, non-code text, and long text. The model is based on character-level input with an input length of 1024, and defines a character set covering 96 characters, specifically including: ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789,;.! ? :'\" / \\|_@#$%^&*~`+-=<>()[]{}. Therefore, the model maps the input text to a 96-dimensional feature vector.
[0102] 2. Model Training: Load the training and test sets. During training, set the learning rate to 0.001 and the batch size to 512, and use the cross-entropy loss function and the Adam optimizer to update the parameters. Finally, derive the model based on the validation loss, which more sensitively reflects the optimization of the model's probability distribution and avoids overfitting.
[0103] III. Classification Prediction
[0104] 1. First, the text to be recognized is read in UTF-8 encoding to obtain the corresponding string. The string is preprocessed to remove redundant spaces and newline characters, and common programming language comments, including single-line comments ( / / ) and multi-line comments ( / ** / ). Simultaneously, the proportion of wide characters and symbols in the string is determined. If the proportion of wide characters or symbols is too high (exceeding a preset ratio), it is judged as not source code and returned to the "other" category. Next, a recognition strategy is adopted based on the character length of the preprocessed string. Four recognition strategies are set, as shown in Table 1 below. Strategy four is used by default, but the algorithm's recognition strategy can also be manually set.
[0105]
[0106] Table 1
[0107] 2. After converting the text into a tensor form that the model can process, input it into the model for inference. For a single text, use softmax to output the probability of each category; for multiple texts, merge the inference results to calculate the average probability distribution. If the maximum probability value exceeds the 50% threshold and the category corresponding to the highest probability belongs to the source code related category, it is determined to be source code; if the probability value is lower than 50%, or the category corresponding to the highest probability belongs to the other or dialog category, it is determined to be non-source code.
[0108] Multiple experiments have demonstrated that this application exhibits superior performance on the test set, achieving an accuracy rate exceeding 95% for all preset categories, including Assembly, C, and C++. In real-world testing scenarios, the model, with its unique design that integrates cross-domain negative samples and character-level classification, can accurately distinguish easily confused non-code content such as Chinese and English documents, effectively avoiding misclassification as source code. Its performance essentially meets the stringent requirements for high-precision source code recognition in production environments, demonstrating practical value in code management, information security testing, and other real-world applications.
[0109] It is evident that existing solutions suffer from several drawbacks. Traditional methods rely on manual feature extraction, which struggles to handle complex scenarios; deep learning-based methods have weak differentiation capabilities for non-code text; and rule-based methods rely on keyword analysis, resulting in low accuracy. Furthermore, existing solutions are generally slow when processing long texts, failing to meet the demands for efficient processing. In contrast, this solution introduces a cross-domain negative sample mechanism, significantly reducing the probability of misclassifying non-code text as source code and greatly improving the accuracy and reliability of source code recognition in real-world scenarios. Simultaneously, a character-length-based recognition strategy is proposed for texts of varying lengths, balancing accuracy and speed.
[0110] In summary, the core steps of the source code identification method for fusing cross-domain negative samples provided in this application are as follows:
[0111] 1. Constructing a sample set: Collect source code samples covering multiple programming languages (such as Assembly, C, C++, Python, etc.) from GitHub, and perform comment removal and text cleaning to standardize the text format; collect diverse English texts, such as news, Yahoo! Answers records, and novel excerpts (such as Harry Potter text), as cross-domain negative samples in the dialog category to simulate easily confused non-code inputs. At the same time, collect Latin, English garbled characters, etc., as general non-code negative samples in the other category to enhance the robustness of the model.
[0112] 2. Model Training: A deep learning-based neural network model (such as the VDCNN model) is built using the PyTorch framework. The dataset is divided into training and test sets and then input into the model. During training, the model optimizes its parameters through backpropagation, learning the deep semantic and structural features of various source code and negative sample data.
[0113] 3. Source code recognition: The input text is processed using a data preprocessing strategy, and the trained model is used to perform inference and prediction on the test text.
[0114] This application trains a source code recognition model by fusing cross-domain negative samples and configures a data preprocessing strategy to process the input text. This not only enhances the model's ability to distinguish non-source code but also balances recognition accuracy with computational efficiency.
[0115] In the above embodiments, a source code recognition method for fusing cross-domain negative samples is provided. Correspondingly, this application also provides a source code recognition device for fusing cross-domain negative samples. The source code recognition device for fusing cross-domain negative samples provided in this application can implement the above-described source code recognition method for fusing cross-domain negative samples. This source code recognition device for fusing cross-domain negative samples can be implemented by software, hardware, or a combination of both. For example, the source code recognition device for fusing cross-domain negative samples may include integrated or separate functional modules or units to perform the corresponding steps in the above methods.
[0116] Please refer to Figure 5 As shown, the source code recognition device 10 that integrates cross-domain negative samples provided in this application includes:
[0117] Preprocessing module 101 is used to preprocess the text to be tested to obtain the target text;
[0118] The source code recognition module 102 is used to input the target text into a pre-trained source code recognition model to obtain the recognition result output by the source code recognition model, wherein the recognition result indicates the category of the text to be tested;
[0119] The source code recognition model is obtained by training the VDCNN model based on a sample set, which includes source code samples and non-source code samples.
[0120] In some embodiments of this application, the apparatus further includes: a model training module for pre-training the source code recognition model in the following manner:
[0121] Source code files in various programming languages were collected from the GitHub open-source platform and organized according to different programming language categories to obtain source code samples;
[0122] Collect English news articles, novel texts, and Q&A records, and label them as dialog; collect English gibberish and Latin texts, and label them as other; the texts in the dialog and other categories constitute non-source code samples; the source code samples and the non-source code samples constitute a sample set;
[0123] The sample set is preprocessed and then divided into a training set and a test set.
[0124] The VDCNN model is trained based on the training set and test set, and the source code recognition model is obtained after the preset training cutoff condition is met.
[0125] In some embodiments of this application, the preprocessing operations include at least one of the following: uniformly encoding the text; removing comments from various types of code using regular expressions; eliminating completely duplicate code files by adopting a deduplication strategy based on file hash values; and excluding cross-language mixed files through syntax analysis.
[0126] In some embodiments of this application, the preprocessing module 101 is specifically used for:
[0127] The text to be tested is read using UTF-8 encoding to obtain the corresponding string;
[0128] The target text is determined based on the character length of the string.
[0129] In some embodiments of this application, the preprocessing module 101 is specifically used for:
[0130] If the character length of the string is less than 2048 characters, then the full text of the string is determined as the target text;
[0131] If the character length of the string is greater than or equal to 2048 characters and less than or equal to 4096 characters, then the first 1024 characters and the last 1024 characters of the string are determined as the target text;
[0132] If the length of the string is greater than 4096 characters, then the first 1024 characters and the last 1024 characters of the string, as well as three randomly sampled text segments from the string, are determined as the target text.
[0133] In some embodiments of this application, the preprocessing module 101 is further configured to determine the proportion of wide characters and symbols in the string before determining the target text based on the character length of the string; if the proportion of wide characters or symbols exceeds a preset ratio, the text to be tested is determined to be non-source code text and returned to the other category.
[0134] The subsequent steps will not be performed.
[0135] The source code identification device for fusion cross-domain negative samples provided in this application embodiment and the source code identification method for fusion cross-domain negative samples provided in this application embodiment are based on the same inventive concept and have the same beneficial effects as the methods they adopt, run or implement.
[0136] This application also provides an electronic device corresponding to the method provided in the foregoing embodiments. The electronic device may be a mobile phone, laptop computer, tablet computer, desktop computer, etc., to execute the above-described method for identifying source code by fusing cross-domain negative samples.
[0137] Please refer to Figure 6 This illustrates a schematic diagram of an electronic device provided by some embodiments of this application. For example... Figure 6 As shown, the electronic device 20 includes: a processor 200, a memory 201, a bus 202, and a communication interface 203. The processor 200, the communication interface 203, and the memory 201 are connected via the bus 202. The memory 201 stores a computer program that can run on the processor 200. When the processor 200 runs the computer program, it executes the source code identification method for fusing cross-domain negative samples provided in any of the foregoing embodiments of this application.
[0138] The memory 201 may include high-speed random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Communication between this system network element and at least one other network element is achieved through at least one communication interface 203 (which can be wired or wireless), such as the Internet, wide area network, local area network, or metropolitan area network.
[0139] Bus 202 can be an ISA bus, PCI bus, or EISA bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. Memory 201 is used to store programs. After receiving an execution instruction, the processor 200 executes the program. The source code identification method for fusing cross-domain negative samples disclosed in any of the foregoing embodiments of this application can be applied to the processor 200, or implemented by the processor 200.
[0140] The processor 200 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of the processor 200 or by instructions in software form. The processor 200 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules may reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory 201. The processor 200 reads the information in memory 201 and, in conjunction with its hardware, completes the steps of the above method.
[0141] The electronic device provided in this application embodiment and the source code identification method for fused cross-domain negative samples provided in this application embodiment are based on the same inventive concept and have the same beneficial effects as the methods they adopt, run or implement.
[0142] This application also provides a computer-readable storage medium corresponding to the source code identification method for fused cross-domain negative samples provided in the foregoing embodiments, wherein a computer program (i.e., a program product) is stored thereon, and the computer program, when run by a processor, executes the source code identification method for fused cross-domain negative samples provided in any of the foregoing embodiments.
[0143] It should be noted that examples of the computer-readable storage medium may also include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical and magnetic storage media, which will not be elaborated here.
[0144] The computer-readable storage medium provided in the above embodiments of this application and the source code identification method for fused cross-domain negative samples provided in the embodiments of this application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the applications stored therein.
[0145] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application, and they should all be covered within the scope of the claims and specification of this application.< / int>
Claims
1. A source code identification method that integrates cross-domain negative samples, characterized in that, include: The text to be tested is preprocessed to obtain the target text; The target text is input into a pre-trained source code recognition model to obtain the recognition result output by the source code recognition model, and the recognition result indicates the category of the text to be tested; The source code recognition model is obtained by training the VDCNN model based on a sample set, which includes source code samples and non-source code samples.
2. The method according to claim 1, characterized in that, The source code recognition model is pre-trained in the following manner: Source code files in various programming languages were collected from the GitHub open-source platform and organized according to different programming language categories to obtain source code samples; Collect English news articles, novel texts, and Q&A records, and label them as dialogs. Collect garbled English text and Latin characters, and label them as "other"; the text of the "dialog" category and the "other" category constitutes the non-source code sample; the source code sample and the non-source code sample constitute the sample set. The sample set is preprocessed and then divided into a training set and a test set. The VDCNN model is trained based on the training set and test set, and the source code recognition model is obtained after the preset training cutoff condition is met.
3. The method according to claim 2, characterized in that, The preprocessing operations include at least one of the following: uniformly encoding the text; removing comments from various types of code using regular expressions; and eliminating completely duplicate code files by adopting a deduplication strategy based on file hash values. Syntax analysis is used to detect and exclude cross-language mixed files.
4. The method according to claim 1, characterized in that, The process of preprocessing the text to be tested to obtain the target text includes: The text to be tested is read using UTF-8 encoding to obtain the corresponding string; The target text is determined based on the character length of the string.
5. The method according to claim 4, characterized in that, Determining the target text based on the character length of the string includes: If the character length of the string is less than 2048 characters, then the full text of the string is determined as the target text; If the character length of the string is greater than or equal to 2048 characters and less than or equal to 4096 characters, then the first 1024 characters and the last 1024 characters of the string are determined as the target text; If the length of the string is greater than 4096 characters, then the first 1024 characters and the last 1024 characters of the string, as well as three randomly sampled text segments from the string, are determined as the target text.
6. The method according to claim 4, characterized in that, Before determining the target text based on the character length of the string, the method further includes: Determine the proportion of wide characters and symbols in the string. If the proportion of wide characters or symbols exceeds a preset value, the text to be tested is determined to be non-source code text and returned to the "other" category. The subsequent steps will not be performed.
7. A source code recognition device that integrates cross-domain negative samples, characterized in that, include: The preprocessing module is used to preprocess the text to be tested to obtain the target text; The source code recognition module is used to input the target text into a pre-trained source code recognition model and obtain the recognition result output by the source code recognition model, wherein the recognition result indicates the category of the text to be tested; The source code recognition model is obtained by training the VDCNN model based on a sample set, which includes source code samples and non-source code samples.
8. The apparatus according to claim 7, characterized in that, The preprocessing module is specifically used for: The text to be tested is read using UTF-8 encoding to obtain the corresponding string; The target text is determined based on the character length of the string.
9. An electronic device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor executes the computer program to implement the method as claimed in any one of claims 1 to 6.
10. A computer-readable storage medium, characterized in that, It stores computer-readable instructions that can be executed by a processor to implement the method as described in any one of claims 1 to 6.