Code data processing method and device, computer equipment and readable storage medium

Abstract

The embodiment of the invention provides a code data processing method and device, computer equipment and a readable storage medium. The method comprises the steps of obtaining a preset code data set; for each piece of preset code sample data, calculating a readability index in combination with the length of each code statement, calculating a character density index according to the proportion of preset type characters, and calculating a code complexity index according to the proportion of preset logic statements so as to calculate a quality score; screening out a plurality of candidate code sample data according to the mass fraction; determining code semantic similarity between a plurality of code statements in each candidate code sample data and corresponding code description information, and determining information entropies of a plurality of grammar structure types contained in the code statements so as to determine a training difficulty score of each candidate code sample data; and determining a plurality of target code sample data based on the training difficulty score of each candidate code sample data to obtain a target code data set. Therefore, the quality of the code data in the code data set can be improved.

Description

Technical Field

[0001] This application relates to the field of data cleaning technology, and in particular to a code data processing method, apparatus, computer equipment, and readable storage medium. Background Technology

[0002] In the pre-training phase of large language models, code data has become crucial for enhancing the model's logical reasoning and program generation capabilities. Training with high-quality code data can significantly improve the performance of large language models in code understanding, completion, conversion, and cross-modal tasks. However, due to the characteristics of code data, such as syntactic rigidity, cross-file dependencies, license infection, and the risk of evaluation leakage, directly using uncleaned raw code data can easily lead to problems such as generating incorrect code, legal compliance risks, and evaluation distortion. Therefore, it is necessary to clean the code data in the code dataset before training the large language model to improve the quality of the code data.

[0003] In related technologies, code data cleaning generally relies on manual rules or simple statistical filtering. Examples include format validation based on regular expressions, duplicate data removal through hash value comparison, filtering based on file extensions or simple keywords, and truncation based on fixed line or character counts. While these methods can remove obviously abnormal samples to some extent, they only perform filtering based on surface features, resulting in relatively low-quality final code data. Summary of the Invention

[0004] This application proposes a code data processing method, apparatus, computer device, and readable storage medium, which can improve the quality of code data in a code dataset.

[0005] To achieve the above objectives, a first aspect of this application proposes a code data processing method, the method comprising: Obtain a preset code dataset, which includes multiple preset code sample data, each preset code sample data containing multiple code statements and code description information for the multiple code statements; For each preset code sample data, a readability index is calculated based on the length of each code statement, a character density index is calculated based on the proportion of preset type characters in the multiple code statements, and a code complexity index is calculated based on the proportion of preset logical statements in the multiple code statements. The corresponding quality score is then calculated by combining the readability index, the character density index, and the code complexity index. Based on the quality score corresponding to each preset code sample data, multiple candidate code sample data are selected from the multiple preset code sample data; Determine the semantic similarity between the multiple code statements in each candidate code sample data and the corresponding code description information, and determine the information entropy of the multiple syntactic structure types contained in each candidate code sample data. Combine the semantic similarity and the information entropy of each candidate code sample data to determine the training difficulty score of each candidate code sample data. Based on the training difficulty score of each candidate code sample data, multiple target code sample data are determined from the multiple candidate code sample data to obtain the target code dataset.

[0006] Accordingly, a second aspect of the embodiments of this application provides a code data processing apparatus, the apparatus comprising: The acquisition module is used to acquire a preset code dataset, which includes multiple preset code sample data, each preset code sample data containing multiple code statements and code description information for the multiple code statements; The calculation module is used to calculate a readability index for each preset code sample data, based on the length of each code statement, to calculate a character density index based on the proportion of preset type characters in the multiple code statements, and to calculate a code complexity index based on the proportion of preset logical statements in the multiple code statements. The module then combines the readability index, the character density index, and the code complexity index to calculate the corresponding quality score. The filtering module is used to filter out multiple candidate code sample data from the multiple preset code sample data according to the quality score corresponding to each preset code sample data; The first determining module is used to determine the code semantic similarity between the plurality of code statements in each candidate code sample data and the corresponding code description information, and to determine the information entropy of the plurality of syntactic structure types contained in each candidate code sample data. Combining the code semantic similarity and the information entropy of each candidate code sample data, the module determines the training difficulty score of each candidate code sample data. The second determining module is used to determine multiple target code sample data from the multiple candidate code sample data based on the training difficulty score of each candidate code sample data, thereby obtaining a target code dataset.

[0007] In some embodiments, the computing module is further configured to: Obtain the first character count for each code statement, and identify code statements whose first character count is greater than a preset character count threshold as target code statements; The number of first statements and the number of second statements containing the target code statements in the preset code sample data are determined, and the readability index of the preset code sample data is obtained based on the ratio between the number of first statements and the number of second statements.

[0008] In some implementations, the preset type characters include letter type characters and number type characters, and the calculation module is further used for: Obtain the total number of characters in the preset code sample data, and the number of second characters corresponding to the letter type characters and the number type characters contained in the preset code sample data; The larger value between the total number of characters and the preset first reference value is selected as the target total number of characters; The character density index of the preset code sample data is calculated based on the ratio between the second character count and the target total character count.

[0009] In some embodiments, the computing module is further configured to: Obtain the number of second statements in the plurality of code statements contained in the preset code sample data, and the number of third statements in the preset logical statements contained therein; Calculate the initial code complexity index based on the ratio between the number of the third statement and the number of the second statement; The larger value between the initial code complexity index and the preset second reference value is selected as the code complexity index of the preset code sample data.

[0010] In some embodiments, the computing module is further configured to: Obtain the first weight corresponding to the readability index and the second weight corresponding to the character density index; Based on the first weight, the readability index is adjusted to obtain the target readability index; Based on the relationship between the character density index and the preset character density threshold, a density identifier value is determined, and the density identifier value is adjusted based on the second weight to obtain the target character density index. Based on the ratio between the code complexity index and the preset benchmark value, an adjustment index value is determined, and the smaller value between the preset third reference value and the adjustment index value is selected as the target code complexity index. Based on a preset first reference value, the difference between the target readability index, the target character density index, and the target code complexity index is calculated sequentially to obtain the quality score corresponding to the preset code sample data.

[0011] In some implementations, the first determining module is further configured to: Obtain the multiple syntax structure types contained in each candidate code sample data, and the frequency of occurrence of each syntax structure type in the multiple code statements; The total frequency of the target occurrence is obtained by summing the occurrence frequencies corresponding to the multiple syntactic structure types. For each syntax structure type, the corresponding type distribution frequency is calculated based on the ratio between the corresponding occurrence frequency and the total occurrence frequency of the target. Based on the frequency distribution of multiple types corresponding to the multiple syntax structure types, the information entropy of the multiple syntax structure types contained in each candidate code sample data is calculated.

[0012] In some implementations, the first determining module is further configured to: For each candidate code sample data, the corresponding semantic understanding difficulty is obtained based on the difference between the preset first reference value and the semantic similarity of the code; Obtain preset third and fourth weights, and adjust the semantic understanding difficulty based on the third weight to obtain the target semantic understanding difficulty; and adjust the information entropy based on the fourth weight to obtain the target information entropy. The training difficulty score for each candidate code sample data is obtained based on the sum of the target semantic understanding difficulty and the target information entropy.

[0013] In some implementations, each preset code sample data also includes task description information corresponding to the plurality of code statements, and the second determining module is further configured to: Based on the task description information of each candidate code sample data, the multiple candidate code sample data are classified to obtain multiple category sets; For each category set, the mean of multiple training difficulty scores corresponding to the multiple candidate code sample data contained therein is calculated to obtain the corresponding average difficulty value; Based on the number of samples of multiple second code samples contained in each category set and the average difficulty value, calculate the number of samples allocated to each category set; For each category set, multiple candidate code sample data are sorted according to the training difficulty score corresponding to each candidate code sample data to obtain a sorting result, and multiple target code sample data are extracted from the sorting result based on the corresponding sample allocation quantity; Based on the target code sample data corresponding to each category set, the target code dataset corresponding to the multiple category sets is obtained.

[0014] In some embodiments, the code data processing apparatus further includes a matching module for: Obtain a preset list of risk interfaces, and match each candidate code sample data with the list of risk interfaces to obtain a first matching result; Based on the first matching result, the multiple candidate code sample data are filtered to obtain multiple filtered candidate code sample data; The license information of each candidate code sample data after filtering is extracted according to a preset regular expression, and the license information is matched with a preset license restriction list to obtain a second matching result; Based on the second matching result, the filtered candidate code sample data is further filtered to obtain the license-filtered candidate code sample data.

[0015] In some embodiments, the code data processing apparatus further includes a comparison module for: For each candidate code sample data, obtain the first semantic vector corresponding to the plurality of code statements and the second semantic vector corresponding to the code description information; Based on the similarity between the first semantic vector and the second semantic vector, a code semantic consistency score is calculated between the plurality of code statements and the code description information; The code semantic consistency score is compared with a preset consistency threshold to obtain the comparison result; Based on the comparison results of each candidate code sample data, the multiple candidate code sample data are filtered to obtain multiple candidate code sample data after semantic filtering.

[0016] Accordingly, a third aspect of the embodiments of this application provides a computer device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the code data processing method of any one of the embodiments of the first aspect of this application.

[0017] Accordingly, a fourth aspect of the embodiments of this application proposes a computer-readable storage medium storing a computer program that, when executed by a processor, implements the code data processing method of any one of the embodiments of the first aspect of this application.

[0018] This application embodiment obtains a preset code dataset, which includes multiple preset code sample data. Each preset code sample data contains multiple code statements and code description information for the multiple code statements. For each preset code sample data, a readability index is calculated based on the length of each code statement, a character density index is calculated based on the proportion of preset type characters in the multiple code statements, and a code complexity index is calculated based on the proportion of preset logical statements in the multiple code statements. A corresponding quality score is calculated by combining the readability index, the character density index, and the code complexity index. Multiple candidate code sample data are selected from the multiple preset code sample data according to the quality score corresponding to each preset code sample data. The semantic similarity between the multiple code statements in each candidate code sample data and the corresponding code description information is determined, and the information entropy of the multiple syntactic structure types contained in each candidate code sample data is determined. A training difficulty score for each candidate code sample data is determined by combining the semantic similarity and the information entropy. Based on the training difficulty score of each candidate code sample data, multiple target code sample data are determined from the multiple candidate code sample data to obtain a target code dataset. Therefore, through multi-dimensional intrinsic code quality assessment and semantic structure analysis, high-quality, semantically consistent, and reasonably distributed target code sample data can be accurately selected from code datasets. Specifically, compared to related technologies that rely on manual rules or simple statistical filtering, this application not only considers the surface features of the code but also deeply evaluates its readability, content density, logical complexity, and description, effectively filtering out low-quality code with poor readability, sparse effective content, or overly simple logic. Simultaneously, by quantifying the complexity and diversity of code structure through semantic similarity and information entropy, it effectively filters out low-quality code samples with small semantic understanding gaps or simple structures, ensuring that the cleaned code data possesses both high quality and training diversity. In summary, this application can improve the quality of code data in code datasets. Attached Figure Description

[0019] Figure 1 This is a schematic diagram of the architecture of the code data processing system provided in the embodiments of this application; Figure 2 This is a flowchart of the code data processing method provided in the embodiments of this application; Figure 3 This is a diagram illustrating the steps for calculating the mass fraction provided in an embodiment of this application; Figure 4 This is a flowchart illustrating the steps for selecting samples based on training difficulty scores provided in an embodiment of this application. Figure 5This is a general flowchart of the code data processing method provided in the embodiments of this application; Figure 6 This is a functional module diagram of the code data processing device provided in the embodiments of this application; Figure 7 This is a schematic diagram of the hardware structure of the computer device provided in the embodiments of this application. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0021] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0022] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0023] In the pre-training phase of large language models, code data has become crucial for enhancing the model's logical reasoning and program generation capabilities. Training with high-quality code data can significantly improve the performance of large language models in code understanding, completion, conversion, and cross-modal tasks. However, due to the characteristics of code data, such as syntactic rigidity, cross-file dependencies, license infection, and the risk of evaluation leakage, directly using uncleaned raw code data can easily lead to problems such as generating incorrect code, legal compliance risks, and evaluation distortion. Therefore, it is necessary to clean the code data in the code dataset before training the large language model to improve the quality of the code data.

[0024] In related technologies, code data cleaning generally relies on manual rules or simple statistical filtering. Examples include format validation based on regular expressions, duplicate data removal through hash value comparison, filtering based on file extensions or simple keywords, and truncation based on fixed line or character counts. While these methods can remove obviously abnormal samples to some extent, they only perform filtering based on surface features, resulting in relatively low-quality final code data.

[0025] Based on this, embodiments of this application provide a code data processing method, apparatus, computer device, and readable storage medium, which can improve the quality of code data in a code dataset.

[0026] The code data processing method, apparatus, computer equipment, and readable storage medium provided in the embodiments of this application are specifically described through the following embodiments. First, the code data processing system in the embodiments of this application is described.

[0027] Please refer to Figure 1 In some embodiments, this application provides a code data processing system, including a terminal 11 and a server 12.

[0028] In some implementations, terminal 11 can be used to acquire, provide, or present code data. For example, it can be a hardware device such as a personal computer, mobile terminal, embedded device, or dedicated data acquisition device; terminal 11 can receive raw code data uploaded by the user or read a preset code dataset from local / network storage through a built-in or external input module, and send the preset code dataset to server 12 to initiate a data processing request.

[0029] In some implementations, the server 12 can be used to execute the core algorithm steps of the code data processing method. For example, it can be a server cluster, a cloud computing platform, a distributed computing system, or a dedicated computing device with a high-performance processor. The server 12 can receive a preset code dataset from the terminal 11, and sequentially perform quality score calculation, candidate code sample data screening, training difficulty score determination, and target code sample data selection to generate a target code dataset, and return the processing results to the terminal 11.

[0030] Terminal 11 and server 12 can connect and interact with each other via wired or wireless communication networks. Terminal 11 is responsible for data collection, uploading and result display, while server 12 is responsible for carrying out and running each step of the data processing method. Together, they complete the cleaning and screening process from raw code data to high-quality target code dataset.

[0031] The code data processing method in this application can be illustrated by the following examples.

[0032] It should be noted that in all specific embodiments of this application, when processing data related to user identity or characteristics, such as user information, user behavior data, user historical data, and user location information, user permission or consent will be obtained first. Furthermore, the collection, use, and processing of this data will comply with relevant laws, regulations, and standards. In addition, when embodiments of this application require access to sensitive personal information of users, separate permission or consent from the user will be obtained through pop-ups or redirects to confirmation pages. Only after obtaining the user's separate permission or consent will the necessary user-related data for the normal operation of the embodiments of this application be obtained.

[0033] In this embodiment, the description will focus on the dimension of a code data processing device, which can be integrated into a computer device. See also Figure 2 , Figure 2 This is a flowchart illustrating the steps of the code data processing method provided in this application embodiment. Taking the code data processing device specifically integrated into a terminal or server as an example, the specific process when the processor on the terminal or server executes the program instructions corresponding to the code data processing method is as follows: Step 101: Obtain a preset code dataset. The preset code dataset includes multiple preset code sample data. Each preset code sample data contains multiple code statements and code description information for the multiple code statements.

[0034] In some implementations, in order to provide a structured, high-quality code data foundation for training large models, a pre-set code dataset can be acquired to construct initial data that supports subsequent in-depth quality assessment and fine-grained screening. This ensures that the input to the cleaning process has the necessary semantic associations and structured features, laying the foundation for improving the overall quality and diversity of the final training data.

[0035] The preset code dataset can be a raw data set consisting of multiple independent code samples, usually from open source code libraries, programming competition platforms or code practice websites, such as JSON format files crawled and initially organized from GitHub repositories or LeetCode problems, which are used as input sources for a series of automated cleaning and evaluation processes.

[0036] The preset code sample data can be a basic data unit in the dataset, which encapsulates information related to a complete code task in a structured form. For example, it is a JSON object containing three fields: task description information (text field), multiple code statements corresponding to the task description information (code field), and code description information for the multiple code statements (description field).

[0037] Among them, code statements can be a set of instructions written in a programming language that can express specific logic or functions corresponding to the task description information, such as Python source code lines that implement the quicksort algorithm, data crawling, or network requests.

[0038] The code description information can be natural language text that is semantically related to the code statements, used to explain the function, purpose or background of multiple code statements, such as a Python implementation of quicksort or a web crawler script for downloading web page images.

[0039] In some implementations, the system can obtain a pre-defined code dataset from a specified data source to initiate the entire code data cleaning and enhancement process. This data source can be a publicly available programming problem and solution platform (such as LeetCode), an open-source code hosting platform (such as GitHub), or a proprietary code practice database. The obtained pre-defined code dataset is typically stored and provided in a structured file format (such as JSON Lines, i.e., .jsonl files) to facilitate line-by-line reading and processing.

[0040] For example, besides obtaining structured (text, code, description) triplet data from standard programming challenge platforms, such datasets can also be automatically built from open-source software repositories. Specifically, projects on platforms like GitHub can be crawled, with the feature summary in the project's README.md file used as text, the content of the core source code files used as code, and a code summarization model (such as a Transformer-based code summarization model) used to automatically generate descriptions for the source code. This significantly expands the sources and scale of training data, covering code samples that more closely resemble real-world development scenarios, which helps train larger code models with stronger generalization capabilities.

[0041] Specifically, the task description information (i.e., the text field) contained in each preset code sample data can be used to represent task instructions or problem description text. For example, it can be a natural language description of a programming problem, such as implementing a function to calculate the greatest common divisor of two non-negative integers.

[0042] The multiple code statements (i.e., the code field) corresponding to the task description information can represent the source code corresponding to the task description information. They are usually composed of multiple code statements (i.e., code lines) that provide a solution to the problem described in the task description information.

[0043] Furthermore, the code description information (i.e., the description field) for multiple code statements can be used to provide a brief textual description or summary of the code function in the code field, and to summarize the overall core logic of multiple code statements.

[0044] In some implementations, to ensure the reliability of the data foundation, basic data integrity checks can be performed first after obtaining the preset code dataset. For example, it can be checked whether each sample simultaneously contains non-empty text, code, and description fields. Samples missing any of these key fields can be filtered out at this stage to prevent invalid data from flowing into subsequent complex calculation processes. For instance, a sample in the format {"text":"sorting algorithm","code":"defsort():pass"} can be discarded directly because it lacks the description field. In this way, obviously incomplete data can be excluded at the beginning of the process, improving overall processing efficiency.

[0045] For example, HTML tags, URL links, leading and trailing whitespace, and sample code data shorter than 100 characters can be removed from a preset code dataset. The removal of HTML tags and URL links is achieved through regular expression matching and replacement.

[0046] Furthermore, language filtering can be performed on each preset code sample data in the preset code dataset, retaining only code samples containing both Chinese and English. This can be achieved by calling the detect interface of the python fasttext_langdetect library.

[0047] In some implementations, samples with fewer than MIN_CODE_LINES (minimum number of lines) or more than MAX_CODE_LINES (maximum number of lines) can be removed from a preset code dataset. For example, the minimum number of lines can be set to MIN_CODE_LINES = 5, and the maximum number of lines can be set to MAX_CODE_LINES = 1000. This effectively filters samples with very small amounts of code, and to some extent, the number of lines measures code completeness.

[0048] A structured, pre-set code dataset lays the foundation for a systematic and automated evaluation of code quality, semantic consistency, and learning difficulty. This ensures that the input to the cleaning process is well-organized and contains rich semantic information, facilitating accurate processing by subsequent filtering and scoring modules.

[0049] Step 102: For each preset code sample data, calculate the readability index based on the length of each code statement, calculate the character density index based on the proportion of preset type characters in multiple code statements, and calculate the code complexity index based on the proportion of preset logical statements in multiple code statements. Combine the readability index, character density index, and code complexity index to calculate the corresponding quality score.

[0050] In some implementations, in order to achieve an objective and quantitative comprehensive evaluation of the quality of code samples from three dimensions—structure, content, and logic—it is possible to combine the proportion of extremely long lines to measure code readability, the proportion of alphanumeric characters to measure the density of effective code content, and the number of logical statements to measure code logic complexity. A unified quality score is then calculated based on weighted rules to establish an automated and measurable code quality filtering standard, thereby providing a core quantitative basis for the subsequent accurate selection of high-quality training samples.

[0051] Among them, readability metrics can be measures used to quantify how visually and structurally the code is easy to read and understand, for example, by counting the proportion of excessively long lines in the code that exceed a certain threshold (such as 500 characters).

[0052] Among them, the preset type character can be a type of key character that is pre-specified when evaluating the validity of the code content. For example, it can include letters (AZ, az) and numbers (0-9), which can be used to reflect the density of substantive logical content (rather than whitespace, comments or special symbols) in the code.

[0053] The character density metric can be a measure of the fullness of effective logical content in multiple code statements. Specifically, it is obtained by calculating the ratio of the total number of characters of a preset type (such as letters and numbers) to the total number of characters in the code (or a comparison benchmark that ensures non-zero values). It can be used to filter out low information density code that contains too many invalid characters (such as redundant spaces and format symbols).

[0054] Among them, the preset logical statements can be specific keywords or statements used to indicate the control flow and logical complexity in multiple code statements, such as the branch and logical operation keywords such as if, for, while, and, or in the Python language.

[0055] Among them, code complexity metrics can be measures used to quantify the complexity of the code's logical structure. For example, it can be obtained by calculating the ratio of the number of preset logical statements (such as branch keywords) to the total number of lines of code (that is, the number of multiple code statements).

[0056] Among them, the mass fraction can be a single quantitative score that synthesizes the evaluation results of multiple dimensions such as readability, content density, and logical complexity. For example, it can be a value between 0 and 1 obtained by combining and deducting the above indicators according to preset weights.

[0057] Please refer to Figure 3 , exemplarily, samples with the number of code lines < MIN_CODE_LINES or the number of code lines > MAX_CODE_LINES can be removed first. Here, the minimum number of lines MIN_CODE_LINES = 5 and the maximum number of lines MAX_CODE_LINES = 1000 can be set. In this way, samples with very little code volume can be filtered, and the code integrity can be measured by the number of lines to a certain extent.

[0058] Please refer to Figure 3 , in some embodiments, a preset character number threshold (MAX_LINE_LEN), such as 500 characters, can be set. Traverse each code statement (i.e., each line) in the code field of the preset code sample data, and obtain the first character number corresponding to this line (i.e., the length of the string of this line). The code statement with the first character number greater than MAX_LINE_LEN is determined as the target code statement (extra-long line).

[0059] Next, count the first statement number (long_lines) of the target code statements in the sample data, and count the second statement number (len_code) of the total code statements included in the code field.

[0060] Finally, based on the ratio between the first statement number and the second statement number, the readability index (denoted as R) of this sample is obtained. The higher this ratio, the worse the code readability. The calculation formula can be expressed as: R = long_lines / len_code.

[0061] Exemplarily, a piece of code has 50 lines (len_code = 50), and among them, 2 lines exceed 500 characters (long_lines = 2), then its readability index R = 2 / 50 = 0.04.

[0062] In this way, the code layout style can be quantified into a specific numerical index, which is used to identify and filter low-quality samples that contain too many long and difficult-to-read code lines.

[0063] In some implementations, when calculating the character density index for each preset code sample data, preset character types can be defined to include alphanumeric characters and numeric characters. The total number of characters in the entire code field string (code_alnum) is obtained, and the total number of alphanumeric characters is counted to obtain the second character count (alnum).

[0064] To avoid division by zero errors or denoting errors when the code is very short, a larger value can be selected from the total number of characters and a preset first reference value (e.g., 1) as the target total number of characters (denominator). This can be achieved using the max(code_alnum,1) function.

[0065] Finally, the character density index (denoted as D) of the sample can be calculated based on the ratio between the number of the second character and the total number of target characters. The calculation formula can be expressed as: D=alnum / max(code_alnum,1).

[0066] For example, if a piece of code has a total of 200 characters (code_alnum=200), of which 160 are letters and numbers (alnum=160), then its character density index D=160 / max(200,1)=0.8.

[0067] In this way, the information density of multiple code statements in each preset code sample data can be quantified, and sparse or inefficient code samples containing too much meaningless whitespace, comments, or special symbols (such as a large number of decorative characters) can be filtered out.

[0068] In some implementations, when calculating the code complexity index for each preset code sample data, a set of keywords corresponding to preset logical statements can be defined, such as {"if", "for", "while", "and", "or", "def", "class", "try", "except"}, etc. These keywords are usually related to control flow, logical judgment and structure definition.

[0069] Furthermore, the number of second statements (len_code) in the code field can be obtained, and the number of third statements (branches) in all code statements that contain the preset logical statement keyword can be counted. Note that a line of code may contain multiple keywords, but this implementation uses line frequency rather than word frequency to count, focusing on the distribution of logical points.

[0070] Furthermore, the ratio of the number of third statements to the number of second statements can be calculated to obtain an initial code complexity index.

[0071] It should be noted that, to ensure this metric has a reasonable lower limit and to avoid it being zero or too small due to very few lines of code or overly simple logic, a larger value can be selected from a preset second reference value (e.g., 0.1) and the initial code complexity metric as the final code complexity metric (denoted as C). That is, C = max(branches / len_code, 0.1).

[0072] For example, a 20-line code (len_code=20) contains the above keyword in 4 lines (branches=4), so its initial complexity is 4 / 20=0.2. Since it is greater than 0.1, the final complexity is C=0.2.

[0073] This allows for the quantification of the logical richness of the code, which helps to balance the dataset in subsequent screening and retain samples with appropriate logical challenges.

[0074] In some implementations, the quality score (denoted as S) can be obtained by weighting and penalizing the three basic indicators mentioned above. This ensures that pre-defined code sample data with poor readability (high R value), low effective content density (low D value), or abnormally complex / simple logic (C value deviating from the norm) should be penalized. Specifically, the quality score is calculated as follows: Managing readability metrics: Obtaining readability metrics The corresponding first weight (w_r), for example, is set to 0.4. This weight is used to adjust R, yielding the target readability metric deduction component (Penalty_R), i.e. .

[0075] Processing character density metrics: Set a character density threshold ( For example, 0.4. (Character density index) Compare with this threshold: If < Then the density label value ( If the value is 1, it is 0; otherwise, it is 1. Obtain the second weight (w_d) corresponding to this identifier value, for example, set it to 0.3. Calculate the deduction component of the target character density index. .

[0076] Handling code complexity metrics: Set a preset baseline value (C_base), for example, 10. Calculate the code complexity metric. The ratio to this baseline value yields the adjustment index value (Adj_C), i.e., Adj_C = C / C_base. A third reference value (S_max), for example 0.3, is set to represent the upper limit of complexity deduction. The smaller value between S_max and Adj_C is selected as the target code complexity index deduction component. .

[0077] Furthermore, the final quality score can be obtained by using a preset first reference value as the benchmark for the full score, such as 1.0. The three deduction factors mentioned above are subtracted sequentially, ensuring the result is non-negative. The calculation formula can be expressed as: Therefore, a comprehensive quality score between 0 and 1 can be obtained, with a higher score indicating a better balance between readability, content density, and complexity.

[0078] Furthermore, the max function can be used to ensure that the quality score is not negative. Specifically: .

[0079] By using the above methods, the quality of code samples can be automatically and standardized to achieve a comprehensive quantitative score based on three key dimensions: code formatting, content validity, and logical complexity. This allows for the objective and efficient identification and filtering of low-quality code samples that are poorly readable, have low information density, or are logically simplistic or chaotic. This lays a solid foundation for subsequent steps that focus on processing high-quality data and improving the overall quality rate of large model training data.

[0080] In some implementations, to automate and objectively quantify code formatting and readability, the number of characters in each code statement can be counted, statements exceeding a preset length threshold can be identified as excessively long lines, and their proportion in the entire code sample can be calculated to obtain a readability index. This transforms the subjective feeling of whether the code layout is clear and easy to read into a concrete and comparable numerical measure, providing a key basis for subsequent comprehensive quality scoring. For example, step 102, "calculating the readability index based on the length of each code statement," may include: (102.a1) Obtain the number of the first character corresponding to each code statement, and determine the code statements whose number of the first character is greater than the preset character number threshold as target code statements; (102.a2) Determine the number of first statements and the number of second statements containing target code statements in the preset code sample data, and obtain the readability index of the preset code sample data based on the ratio between the number of first statements and the number of second statements.

[0081] The first character count can be the total number of characters contained in a single code statement, such as the number of characters in a line of Python code `def quicksort(arr):`. This is the basic data used to determine whether the line of code is too long and exceeds the normal readability range.

[0082] The preset character count threshold can be a pre-defined standard value for the number of characters used to define whether a line of code is considered excessively long, such as 400 or 500 characters. When the number of characters in a single code statement exceeds this threshold, it is considered to potentially affect readability. The setting of this threshold balances the distinction between normal code length and abnormally long code.

[0083] The target code statement can be a line of code whose number of characters exceeds a preset character count threshold. For example, it can be a line of code containing extremely long string concatenations or complex nested expressions.

[0084] The first statement count can be the total number of lines identified as target code statements (i.e., excessively long lines) in a preset code sample data.

[0085] The second statement count can be the total number of lines of code statements contained in a preset code sample data, that is, the overall size of the code segment, which is the baseline value of the denominator when calculating the proportion of super long lines.

[0086] For example, a reasonable preset character limit (MAX_LINE_LEN) can be set. This limit can be determined based on the default display width of a common code editor, programming style guidelines (such as PEP 8's recommendations on Python code line length), or empirical values, such as 120, 200, or 500 characters. This embodiment uses 500 characters as an example for illustration.

[0087] Furthermore, for multiple code statements in the preset code sample data, if they were not previously split line by line, they can be split into multiple independent code statements based on newline characters. Then, each code statement is traversed, and the number of the first character corresponding to each code statement is obtained by calling a string length calculation function (such as the len() function in Python), that is, the total number of all characters (including letters, numbers, symbols, spaces, etc.) contained in that line of code.

[0088] Next, the number of the first character of each code statement can be compared with a preset character count threshold. All code statements with a first character count greater than the preset character count threshold are identified as target code statements, i.e., excessively long lines.

[0089] For example, if the first character of a code statement is 80 and the preset character count threshold is 500, it will not be identified as a target code statement; however, a line of code containing extremely long string literals or unformatted JSON data may have more than 500 characters and will be identified as a target code statement.

[0090] Furthermore, after identifying all target code statements in the current preset code sample data, the readability metric can be calculated through the following process: First, count the total number of target code statements identified in the preset code sample data to obtain the first statement count, denoted as `long_lines`. Simultaneously, obtain the total number of code statement lines contained in the `code` field of this sample, i.e., the second statement count, denoted as `len_code`. The total number of lines here includes all non-empty lines and empty lines (depending on the specific implementation definition, purely empty lines or lines containing only comments can be ignored to focus more on the logic code).

[0091] Furthermore, the readability index of the preset code sample data can be calculated based on the ratio between the number of first statements and the number of second statements. The calculation formula is: . The value of is between [0,1]. The closer the value is to 1, the higher the proportion of excessively long lines and the worse the readability of the code. The closer the value is to 0, the better the control of the code line length and the better the basic readability.

[0092] By using the above methods, subjective judgments about code formatting style can be transformed into an objective and calculable numerical indicator. This indicator can effectively identify samples from a large amount of code data that suffer from poor readability due to excessively long lines, providing crucial input for subsequent comprehensive screening based on quality scores.

[0093] In some implementations, the preset character types include alphanumeric characters and numeric characters. To objectively assess the information density of substantive logical content (relative to formatting characters, comments, and other invalid content) in a code sample, the total number of letters and numbers in the code can be counted, and their proportion can be calculated based on a zero-prevention benchmark (the larger of the total number of characters and a preset reference value) to obtain a character density index. This provides a key criterion for identifying and filtering low-quality code with sparse information. For example, step 102, "calculating the character density index based on the proportion of preset character types in multiple code statements," may include: (102.b1) Obtain the total number of characters in the preset code sample data, and the number of second characters corresponding to alphanumeric characters in the preset code sample data; (102.b2) Select the larger value between the total number of characters and the preset first reference value as the target total number of characters; (102.b3) Calculate the character density index of the preset code sample data based on the ratio between the number of second characters and the total number of target characters.

[0094] The total number of characters can be the total number of all characters in the code string contained in a preset code sample data, including letters, numbers, punctuation, spaces and special symbols, and is a basic indicator for measuring the original size of the code sample.

[0095] Among them, alphabetic characters can be all characters in the code string that belong to the alphabetic category, usually including uppercase English letters (AZ) and lowercase English letters (az), which are the core character set that constitutes logical entities such as programming language keywords, variable names, and function names.

[0096] Among them, numeric type characters can be all characters in the code string that belong to the numeric category, namely Arabic numerals 0 to 9, which often appear in constant values, indices, or identifiers.

[0097] The second character count can be the sum of the occurrences of all alphanumeric characters in a preset code sample data, used to reflect the number of core characters in multiple code statements that express substantive logic and data content.

[0098] The first reference value can be a preset positive integer constant (e.g., 1) to avoid division by zero errors or calculation failures when the code is empty or extremely short, thus enhancing the robustness of the calculation.

[0099] The target total number of characters can be the denominator value used to calculate the character density index. It is determined by taking the larger value between the total number of characters and the first reference value, which represents a valid benchmark for the total number of characters protected against zero.

[0100] It's important to note that high-quality code should focus on expressing logic and algorithms, and its character composition should primarily consist of letters and numbers with actual semantic meaning. An excessively high proportion of whitespace, comments, and special characters (such as numerous parentheses, commas, and decorative characters) may indicate a loose code structure, the presence of meaningless content, or traces of automatic generation, which negatively impacts the code's information density and learning value. Therefore, the required number of characters can be obtained through the following process: First, for a given code sample data, treat the entire content of its `code` field as a single, complete string. Calculate the length of this string to obtain the total number of characters (denoted as ). This number includes all characters in the code, such as letters, numbers, spaces, newlines, tabs, various punctuation marks, and special characters.

[0101] Furthermore, the number of characters of a preset type can be counted. In this embodiment, the preset type characters may include alphanumeric characters (including uppercase and lowercase English letters, and possibly other letters if multilingualism is considered) and numeric characters (0-9). The total number of these two types of characters is counted to obtain the second character count (denoted as...). In practice, this can be achieved by iterating through the code string and using built-in character classification functions of the programming language (such as Python's str.isalpha() and str.isdigit() methods) to determine and count characters.

[0102] Specifically, before calculating the ratio, two boundary cases need to be considered: one is that the code may be extremely short (e.g., only 1-2 characters), in which case... The value is very small, and the calculated ratio may fluctuate greatly, lacking statistical significance; secondly... Theoretically, it could be 0 (empty code), which would lead to a division error.

[0103] To address the aforementioned issue, a preset first reference value is introduced, typically set to 1, as a safety lower bound for the denominator. During calculation, this can be determined from the total number of characters. Between the preset first reference value (e.g., 1), the larger one is selected using the maximum value function (max) as the target total number of characters. This ensures the numerical stability of the character density index calculation process.

[0104] Furthermore, after obtaining a stable denominator, the character density index can be calculated through the following process. : Count the second character Total number of characters in the target Divide them to get the ratio, which is the character density index (denoted as D). The calculation formula is: .

[0105] in, The value of is between [0, 1]. The closer its value is to 1, the more likely the code consists of letters and numbers, with high effective information density, and the code is likely to be very compact and concise. The lower its value, the higher the proportion of non-alphanumeric characters (such as spaces, newlines, comment characters, and special operators) in the code, and the code may be sparse or contain a large amount of non-core content.

[0106] By using the above methods, the density of alphanumeric characters carrying the core logic in the code can be accurately quantified. This can effectively identify code with low information density due to excessive whitespace, comments, or unnecessary symbols, thus providing key quantitative evidence for the completeness and effectiveness of the code content in subsequent comprehensive quality scoring, and helping to select high-quality training samples that are information-rich and logically compact.

[0107] In some implementations, to objectively quantify the complexity of code logic structure and avoid filtering out low-value training samples with overly simple or abnormally complex logic, a preliminary complexity can be calculated by statistically analyzing the proportion of control flow and logical keywords (such as if, for, etc.) in the total number of lines of code. A minimum threshold is then set to ensure the effectiveness of the indicator, thereby transforming the code logic into a stable, comparable metric that reflects the true logic density, providing a key dimension for quality assessment. For example, step 102, "calculating the code complexity indicator based on the proportion of preset logical statements in multiple code statements," may include: (102.c1) Obtain the number of second statements in the preset code sample data and the number of third statements in the preset logical statements contained therein; (102.c2) Calculate the initial code complexity index based on the ratio between the number of third statements and the number of second statements; (102.c3) Select the larger value from the initial code complexity index and the preset second reference value as the code complexity index of the preset code sample data.

[0108] The third statement count can be the total number of times all statements belonging to preset logical statements (such as if, for, while, and, or branches and logical operation keywords) appear in a preset code sample data. This count directly reflects the complexity of the control flow and conditional logic in the code.

[0109] The initial code complexity metric can be obtained by calculating the ratio of the number of third statements (number of logical keywords) to the number of second statements (total number of lines of code).

[0110] The second reference value can be a preset small positive constant (e.g., 0.1) used to compare with the initial code complexity index and take the larger value to ensure that the final code complexity index is not lower than this value, thereby avoiding the evaluation distortion caused by the index value being too low when there are many lines of code but very few logical statements.

[0111] For example, the total number of lines contained in the `code` field of a preset code sample data, i.e., the number of second statements, can be obtained and denoted as `len_code`. This count can include all non-empty lines.

[0112] Secondly, it's necessary to count the frequency of predefined logical statements. These predefined logical statements typically correspond to keywords or syntactic structures in programming languages ​​used to implement control flow and logical operations. For example, in Python, a list of keywords can be defined, such as {"if", "elif", "else", "for", "while", "try", "except", "finally", "and", "or", "def", "class"}. These keywords represent conditional branches, loops, exception handling, logical operations, and function / class definitions, respectively, and are the core elements constituting the program's logical skeleton.

[0113] Furthermore, each line of code in the `code` field can be iterated through to check if any of the preset logical statement keywords from the list appear in that line. The number of lines containing at least one such keyword is counted to obtain the number of third statements, denoted as `branches`. Note that this embodiment counts on a "line" basis; that is, if a line contains at least one relevant keyword, it is counted as 1, without accumulating the number of times it appears repeatedly within a line.

[0114] For example, a line of code "ifx>0 and y<10:" contains both the keywords "if" and "and", but in this step it is counted as only one line of logical statement.

[0115] This allows us to obtain the overall size of the code (len_code) and the number of statements (branches) that carry the core logic structure, providing basic data for the computational complexity ratio.

[0116] Furthermore, after obtaining the statistical data, the initial indicators are calculated through the following process: Divide the number of third statements (branches) by the number of second statements (len_code) to obtain a ratio. This ratio reflects the proportion of all lines of code used to implement the core logic structure. This ratio is the initial code complexity metric (branches / len_code).

[0117] In some implementations, considering that the initial code complexity metric might be overly sensitive or lose discriminative power in boundary cases (e.g., for extremely short code, len_code is very small, and the initial code complexity metric might be 0 or 1), a preset second reference value (e.g., 0.1) can be set to characterize the minimum basic logical complexity that the preset code sample data should possess. Then, the larger of the initial code complexity metric and the preset second reference value is selected using a maximum value function (max) as the final code complexity metric (denoted as cc) for the preset code sample data. The specific formula is as follows: ; The closer cc is to 1, the more likely that almost every line of code involves logical control, and the code may be very compact with tightly interwoven logic; the lower cc is, the smaller the proportion of lines in the code that directly express logic, and the more likely it contains a large number of simple assignment, print statements or blank lines.

[0118] For example, a simple "Hello World" program or a script with only continuous assignment operations might have a cc value of 0 or close to 0. However, a function implementing quicksort or complex business rules will typically have a significantly higher cc value. Thus, cc provides a preliminary and direct measure of code logic density.

[0119] By using the above methods, the density of control flow and logic structure in the code can be accurately quantified, and a safety net mechanism can be introduced to ensure the effectiveness of the measurement. In this way, it is possible to objectively distinguish between code samples with overly sparse logic (low training value) and those with moderately complex logic, thereby providing a stable and reliable input on the logic complexity of the code for subsequent comprehensive quality scoring, and ensuring that the code selected in the end has sufficient training value and representativeness at the logical level.

[0120] In some implementations, to integrate the independent evaluations of readability, content density, and logical complexity into a unified comprehensive quality score, differentiated weighting and transformation rules (such as direct weighting of readability, setting thresholds to trigger deductions for density, and normalizing and constraining complexity) are used. This establishes an automated and refined evaluation standard that simultaneously reflects code formatting compliance, content richness, and logical rationality, providing a single, clear, and operable quantitative basis for judging code quality. For example, step 102, "combining readability metrics, character density metrics, and code complexity metrics to calculate the corresponding quality score," may include: (102.d1) Obtain the first weight corresponding to the readability index and the second weight corresponding to the character density index; (102.d2) Based on the first weight, the readability index is adjusted to obtain the target readability index; (102.d3) Based on the relationship between the character density index and the preset character density threshold, the density identifier value is determined, and the density identifier value is adjusted based on the second weight to obtain the target character density index; (102.d4) Based on the ratio between the code complexity index and the preset benchmark value, determine the adjustment index value, and select the smaller value between the preset third reference value and the adjustment index value as the target code complexity index; (102.d5) Based on the preset first reference value, the difference is calculated with the target readability index, the target character density index and the target code complexity index in sequence to obtain the quality score corresponding to the preset code sample data.

[0121] The first weight can be a preset multiplication coefficient (e.g., 0.004) used to adjust the readability index (i.e., the proportion of excessively long lines) to control the degree of influence of readability issues in the final quality score calculation.

[0122] The second weight can be a preset penalty coefficient (e.g., 0.3) used to weight the density flag value triggered by insufficient character density, in order to determine the fixed percentage that should be deducted from the quality score for the defect of insufficient content density.

[0123] The target readability index can be an adjusted value obtained by multiplying the original readability index (proportion of excessively long lines) by the corresponding first weight. This value directly represents the number of points that should be deducted from the base score due to readability issues (excessively long lines).

[0124] The character density threshold can be a preset standard value (e.g., 0.4 or 40%) used to judge whether the code content is sufficiently substantial.

[0125] The density flag value can be a binary flag (e.g., 1 if below the threshold, 0 otherwise) determined by comparing the character density index with the character density threshold, used to indicate whether a penalty needs to be imposed for the deficiency of insufficient content density.

[0126] The target character density index can be an adjusted value obtained by multiplying the density identifier value by the corresponding second weight.

[0127] The adjustment index value can be an intermediate value obtained by dividing the code complexity index by a preset baseline value (e.g., 10).

[0128] The target code complexity index can be a smaller value selected from the adjusted index value and a preset third reference value (e.g., 0.3). Its function is to limit the deduction caused by excessive logic complexity to a preset upper limit, and prevent single-dimensional defects from having an excessive impact on the total score.

[0129] In some implementations, the readability metric R, character density metric D, and code complexity metric C can be combined into a single quality score S to reflect the overall quality level of the sample.

[0130] Furthermore, the weights of each deduction can be determined. Specifically, the first weight (denoted as w_r) corresponds to the deduction intensity for poor readability and can be set to 0.4; the second weight (denoted as w_d) corresponds to the deduction intensity for insufficient character density and can be set to 0.3. That is to say, every 1% of readability issues (proportion of excessively long lines) will result in a 0.4% reduction in the total score; while once the character density falls below the threshold, a larger fixed deduction (0.3 points, or 30%) will be triggered directly. The specific weights can be set according to the actual situation.

[0131] Furthermore, readability metrics can be weighted based on the first weight. Adjustments can be made to obtain the target readability index. .

[0132] Furthermore, a preset character density threshold can be set (denoted as...). For example, 0.4. The calculated character density index... and A comparison is then made. Based on the comparison results, a density identifier value (denoted as ) is determined. ):like ,but A value of 1 indicates that the character density does not meet the standard; otherwise, A value of 0 indicates that the standard has been met.

[0133] Furthermore, a second weight w_d (e.g., 0.3) can be used to adjust the density identifier value to obtain the target character density index. .

[0134] In some approaches, the code complexity metric `cc` can be transformed into a capped penalty. Specifically, a preset baseline value, such as 10, can be set to standardize the complexity metric `cc` onto a larger scale, thereby controlling its impact on the final score. For example, an adjusted metric value could be calculated. .

[0135] Understandably, since cc itself is usually between 0.1 and a few tenths of a percent, after dividing by a preset baseline value (such as 10), This is usually a small number much less than 1 (such as 0.01-0.05), which represents the basic deduction rate due to complexity.

[0136] Furthermore, a preset third reference value, such as 0.3, can be set, representing the upper limit of the complexity penalty. The difference between the third reference value and the preset baseline value is used as the target code complexity indicator, i.e. This achieves a non-linear penalty for code complexity: within a reasonable range, higher complexity results in a greater deduction; however, the penalty has an upper limit to prevent excessive elimination due to overly complex single samples, while also controlling its maximum impact on the total score.

[0137] Finally, the final score is synthesized by successively deducting the three penalties mentioned above from the perfect score. Setting a preset first reference value as the benchmark for the perfect score, for example, 1.0, the formula for calculating the quality score can be: ; Furthermore, By using the max(...,0) operation, we can ensure that the quality score will not be negative, and the minimum value is 0.

[0138] By using the above methods, multiple independent indicators reflecting quality issues at different levels of the code can be integrated into a comprehensive and balanced single quality score through weighting, threshold triggering, and upper limit constraint rules. This enables a refined and automated comprehensive evaluation of the overall quality of the code sample, and provides a direct and reliable decision-making basis for subsequent steps to efficiently and accurately select high-quality candidate code samples based on clear score thresholds.

[0139] Step 103: Select multiple candidate code sample data from multiple preset code sample data according to the quality score corresponding to each preset code sample data.

[0140] In some implementations, in order to transform the results of the preliminary multidimensional quality assessment into actual screening operations and thus construct a cleaner, higher-quality subset of data, the comprehensive quality score calculated for each preset code sample data can be compared with a preset quality threshold. Samples with scores that meet the standard (e.g., not lower than 0.5) can be selected as candidate code sample data. This transforms subjective and fragmented quality perception into objective and batch automated filtering behavior, providing a high-quality input data foundation for subsequent more refined semantic analysis and difficulty assessment.

[0141] Among them, the candidate code sample data can be those individual code data units that meet the quality standards after being filtered from the preset code dataset through a comprehensive quality score calculated based on dimensions such as readability, content density, and logical complexity. For example, samples with a quality score greater than or equal to 0.5.

[0142] Exemplarily, a clear quality score threshold (denoted as T) can be set. This threshold is determined based on empirical judgment of code quality or tuning on a small-scale validation set. For example, T = 0.5 can be set, that is, samples with a quality score lower than 0.5 are considered low-quality code. Then, all preset code sample data are traversed, and their corresponding quality scores score are compared with the preset threshold T. For each preset code sample data, if its quality score score is greater than or equal to the threshold T (i.e., score >= T), the sample is retained and determined as candidate code sample data; if score < T, the sample is filtered out.

[0143] Exemplarily, assume that the preset code data set contains 10,000 samples. After calculation, 6,500 samples have score >= 0.5, and these 6,500 samples are screened as candidate code sample data and enter the subsequent processing flow.

[0144] In this way, low-quality samples that perform significantly poorly in terms of readability, character density, and complexity balance can be quickly and batch removed, significantly improving the overall baseline quality of the retained data, and providing a purer input for subsequent computationally intensive deep processing (such as semantic vectorization, AST parsing).

[0145] Step 104, determine the code semantic similarity between multiple code statements in each candidate code sample data and the corresponding code description information, and determine the information entropy of multiple syntax structure types included in each candidate code sample data. Combine the code semantic similarity and information entropy of each candidate code sample data to determine the training difficulty score of each candidate code sample data.

[0146] In some embodiments, in order to comprehensively evaluate the potential learning difficulty of code samples from two dimensions of semantic understanding and syntax structure, and provide a basis for constructing a training data set with a balanced difficulty distribution, the semantic understanding gap can be measured by calculating the semantic similarity between the code and its description text, and at the same time, the abstract syntax tree of the code is parsed and the information entropy of the node type distribution is calculated to measure the syntax structure complexity. The evaluation results of these two dimensions are weighted and fused into a comprehensive training difficulty score, so as to quantify the difficulty level of code samples into an objective index that can be used for subsequent data quota.

[0147] Among them, the code semantic similarity can be a metric value used to quantify the degree of consistency in semantic meaning between code statements and their corresponding description information. For example, it is obtained by calculating the dot product or cosine similarity after converting the two into semantic vectors through a pre-trained language model. The higher this value, the more matching the code and the description, and the more direct the semantic understanding.

[0148] Among them, the syntax structure type can be the specific syntax element category represented by each node in the Abstract Syntax Tree (AST) generated by parsing the code, such as function definition, loop statement, conditional expression, assignment operation, etc., which is the basic unit for analyzing the syntax structure of the code.

[0149] Information entropy can be calculated by statistically analyzing the frequency of all syntactic structure types in a code sample and using the information entropy formula. The higher the entropy value, the richer and more diverse the syntactic structure types in the code, the more evenly distributed they are, and the higher the structural complexity.

[0150] Among them, the training difficulty score can be a single score that combines two dimensions: code semantic similarity and syntactic structure information entropy, used to characterize the difficulty of learning and mastering the candidate code sample data for the model.

[0151] For example, a pre-trained sentence vector model can be used to map code and descriptive text into vectors (i.e., embeddings) in a high-dimensional semantic space. For instance, paraphrase-multilingual-MiniLM-L12-v2. This model can encode sentences in any language into dense 384-dimensional vectors and has good representation capabilities for both code text and natural language descriptions.

[0152] Next, for a candidate code sample, its multiple code statements (code field) and their code description information (description field) are input into the sentence vector model to obtain the corresponding first semantic vector. Second semantic vector .

[0153] Then, calculate and Cosine similarity between transposes of is used as semantic similarity of code. The calculation process is as follows: ; Therefore, it is possible to... The lower the similarity, the greater the semantic clarity and alignment quality of the quantified samples. This means that the model needs to bridge a larger semantic gap to establish a connection between the code and the description, and the learning difficulty increases accordingly.

[0154] In some implementations, the `code` field of candidate code sample data can be parsed into an Abstract Syntax Tree (AST). This can be achieved using a language-specific parser library; for example, for Python code, the `tree_sitter` library and its Python syntax parser can be used.

[0155] Furthermore, a depth-first traversal of the AST can be performed to extract the type (i.e., syntax structure type) of all nodes in the tree, such as FunctionDef, If, For, While, Assign, Call, BinOp, Name, Constant, etc. The frequency of each syntax structure type can then be counted.

[0156] Then, based on these frequencies, the information entropy (Entropy) of the distribution of syntactic structure types is calculated to measure the irregularity of the syntactic structures. Suppose there are M different syntactic structure types, and the frequency (CNT) of each syntactic structure type is counted, along with the total frequency of all syntactic structure types. Given the sum of frequencies of all syntax structure types, the information entropy of the candidate code sample data is... The calculation can be performed in the following ways: ; The above summation calculation applies to all M syntax structure types.

[0157] For example, a piece of code containing only consecutive assignment statements might have only a few node types, such as Assign and Name, resulting in a low entropy value H. In contrast, a complex algorithm containing function definitions, conditional branches, loops, and various expressions has diverse node types that are relatively evenly distributed, leading to a significantly higher entropy value H. High entropy usually indicates a more complex code structure that is more difficult to understand and imitate, thus suggesting a higher learning difficulty.

[0158] In some implementations, training difficulty scoring This can be obtained by combining and converting the two indicators mentioned above. The specific calculation process is as follows: ; in, To use code semantic similarity The semantic understanding difficulty obtained from the conversion is as follows: ; For the initial information entropy The information entropy obtained by normalization; The preset adjustment weights can be set according to the actual situation.

[0159] In this way, a quantitative score for training difficulty can be obtained. This score comprehensively reflects the challenge of the candidate code sample data in both semantic alignment and syntactic structure. A higher score indicates that the code sample may be more difficult for the model to learn, thus promoting the overall development of the model's capabilities.

[0160] In some implementations, to objectively quantify the complexity and diversity of code syntax structure and provide an accurate basis for assessing the difficulty of learning the code's syntax, all syntax node types can be obtained by parsing the code, counting the occurrence frequency of each type, and calculating its distribution probability in the entire node set. Then, the uncertainty of this distribution is calculated based on the information entropy formula, thereby transforming the inherent syntactic structure features of the code into a standardized information entropy value that reflects its structural complexity. For example, step 104, "determining the information entropy of multiple syntax structure types contained in each candidate code sample data," may include: (104.a1) Obtain the multiple syntax structure types contained in each candidate code sample data, and the frequency of each syntax structure type in multiple code statements; (104.a2) The total frequency of the target occurrence is obtained by summing the occurrence frequencies corresponding to multiple syntactic structure types; (104.a3) For each syntactic structure type, the corresponding type distribution frequency is calculated based on the ratio between the corresponding occurrence frequency and the total occurrence frequency of the target; (104.a4) Based on the frequency of multiple type distributions corresponding to multiple syntactic structure types, calculate the information entropy of multiple syntactic structure types contained in each candidate code sample data.

[0161] The frequency of occurrence can be the total number of times a specific syntax structure type (such as a function definition node or a conditional judgment node) appears in an abstract syntax tree generated by parsing the code in a candidate code sample data. It is the basic data reflecting the frequency of use of that syntax element in the code.

[0162] The total frequency of the target occurrence can be the sum of the occurrence frequencies of all identified syntax structure types in the abstract syntax tree corresponding to a candidate code sample data.

[0163] Among them, the frequency of type distribution can be the ratio of the occurrence frequency of a certain syntax structure type to the total occurrence frequency of the target, which is used to characterize the proportion of that syntax structure type in all syntax components in the current candidate code sample data.

[0164] In some implementations, the more diverse and evenly distributed the syntax node types used in the candidate code sample data, the more complex and unpredictable its structure, and the higher the corresponding information entropy value.

[0165] For example, the `code` field (multiple code statements) in candidate code sample data can be parsed to generate its corresponding AST tree. For instance, for Python code, the `tree_sitter` library and its Python language parser component can be used. Specifically, a parser object can be initialized, the language type set to 'python', and then the `parser.parse(bytes(code,'utf-8'))` method can be called to parse the code string into an AST tree.

[0166] Furthermore, the generated AST tree can be traversed using either depth-first or breadth-first search to visit every node in the tree. For each visited node, its node type is extracted. This type is an identifier defined by the parser that represents a specific syntax structure type, such as FunctionDef (function definition), If (conditional statement), For (loop statement), Assign (assignment statement), Call (function call), BinOp (binary operation), Name (variable name), Constant (constant), etc.

[0167] During the traversal, for each node type encountered, the frequency of its corresponding syntax structure type is incremented by 1. After the traversal is complete, a list of all syntax structure types contained in the candidate code sample data, as well as the frequency of each type in the entire AST, can be obtained.

[0168] Furthermore, the occurrence frequencies of all statistically obtained syntax structure types can be summed to obtain the total frequency of the target occurrence (denoted as N). For example, if the syntax structure types FunctionDef, arguments, Return, and BinOp appear once in the candidate code sample data, and Name appears twice, then the total frequency of the target occurrence is... For: 1 (FunctionDef)+1 (arguments)+1 (Return)+1 (BinOp)+2 (Name)=6.

[0169] Furthermore, to measure the dispersion of each syntactic structure type in the candidate code sample data, absolute frequency can be converted into relative proportion. The specific process is as follows: For each identified grammatical structure type, its frequency v is compared with the total frequency of the target occurrence. Dividing by the other yields the type distribution frequency for that type. Then, for each syntax structure type, the corresponding distribution dispersion value is further calculated using the following method. : ; Therefore, the information entropy of each candidate code sample data can be calculated by summing the distribution dispersion values ​​corresponding to multiple syntactic structure types. The specific process is as follows: ; Furthermore, the information entropy can be normalized to the range of 0-1 to obtain the normalized information entropy norm_ent. For example, if the information entropy is 8.0, then norm_ent = min(entropy / 8.0, 1.0), where min is the minimum function, representing the minimum value between the two.

[0170] By using the above methods, the diverse syntactic structures in the code can be transformed into a single information entropy value that reflects the uniformity and complexity of its distribution. In this way, we can go beyond the statistics of simple keywords or line counts and accurately measure the inherent complexity of the code syntax from an information theory perspective. This provides a core, objective, and comparable quantitative input on the syntactic dimension for subsequent comprehensive training difficulty scoring, helping to distinguish between code samples with simple structures and those with complex and diverse structures.

[0171] In some implementations, to normalize and merge the code semantic similarity and information entropy, which respectively reflect the semantic understanding gap and syntactic structural complexity, into a unified comprehensive score, the semantic similarity can be converted into a complementary understanding difficulty. These are then weighted and adjusted with specific weights for information entropy before being added together to obtain a training difficulty score that balances the contributions of both semantic and syntactic dimensions and can be directly used for comparison and sampling. This provides a comprehensive and operable unified standard for quantifying the overall learning difficulty of code samples. For example, step 104, "combining the code semantic similarity and information entropy of each candidate code sample data to determine the training difficulty score of each candidate code sample data," may include: (104.b1) For each candidate code sample data, the corresponding semantic understanding difficulty is obtained based on the difference between the preset first reference value and the code semantic similarity. (104.b2) Obtain the preset third weight and fourth weight, and adjust the semantic understanding difficulty based on the third weight to obtain the target semantic understanding difficulty, and adjust the information entropy based on the fourth weight to obtain the target information entropy; (104.b3) The training difficulty score of each candidate code sample data is obtained based on the sum of the target semantic understanding difficulty and the target information entropy.

[0172] The semantic understanding difficulty can be a value obtained by subtracting the code semantic similarity from a preset full score benchmark (e.g., 1), which is used to represent the quantitative value of the model's difficulty in understanding the semantics of the current code statement.

[0173] The third weight can be a preset multiplication coefficient (e.g., 0.5) used to adjust the weighting of semantic understanding difficulty, so as to control the proportion of the difficulty of semantic understanding dimension in the final comprehensive score.

[0174] The fourth weight can be a preset multiplication coefficient (e.g., 0.5), which, when combined with the third weight, equals 1. It is used to weight and adjust the normalized information entropy to control the proportion of syntactic complexity in the final overall score.

[0175] The target semantic understanding difficulty can be an adjusted value obtained by multiplying the semantic understanding difficulty by the corresponding "third weight". This value represents the semantic dimension difficulty component that is finally included in the total score after weight adjustment.

[0176] The target information entropy can be an adjusted value obtained by multiplying the normalized information entropy by the corresponding fourth weight. This value represents the grammatical structure complexity component that is finally included in the total score after weight adjustment.

[0177] In some implementations, code semantic similarity can be transformed into an indicator positively correlated with difficulty, requiring a reversal process. For example, a first reference value of 1 can be set, representing the ideal situation of complete semantic similarity (i.e., perfect similarity). Then, the semantic similarity between this first reference value and the code can be calculated. The difference between them yields the semantic understanding difficulty. The calculation process is as follows: .

[0178] because The theoretical range of cosine similarity is [-1, 1], therefore The range of its value is [0,2]. The lower, The higher the value, the greater the semantic gap and the more difficult it is to understand.

[0179] Furthermore, the semantic understanding difficulty and information entropy can be scaled and weighted to adjust their relative importance in the final difficulty score. Specifically, the third weight can be set to 0.4 and the fourth weight to 0.6. The weights can also be adjusted based on the emphasis placed on different sources of difficulty, such as setting the third weight to 0.7 and the fourth weight to 0.3, etc.

[0180] Furthermore, the semantic understanding difficulty can be multiplied by the third weight to obtain the target semantic understanding difficulty. For example, if the semantic understanding difficulty is 0.15 and the third weight is 0.5, then the target semantic understanding difficulty is 0.5. 0.15 = 0.075.

[0181] Furthermore, the information entropy can be multiplied by the fourth weight to obtain the target information entropy. For example, if the information entropy is 0.4375 and the fourth weight is 0.5, then the target information entropy is 0.5. 0.4375≈0.21875.

[0182] Subsequently, a training difficulty score can be obtained for each candidate code sample data based on the sum of the target semantic understanding difficulty and the target information entropy. For example, using the above example, the training difficulty score = 0.075 + 0.21875 = 0.29375. This score considers both the difficulty of bridging the semantic gap and the difficulty of parsing complex syntactic structures. Samples with high scores mean that the model needs to cope with both ambiguous instruction-code correspondences and complex internal code structures, making learning more challenging; samples with low scores are relatively easier to master. This serves as the core basis for subsequent dynamic quota sampling based on topics and difficulty, and is crucial for constructing a high-quality dataset with a balanced difficulty distribution that allows for gradual model training.

[0183] Step 105: Based on the training difficulty score of each candidate code sample data, determine multiple target code sample data from multiple candidate code sample data to obtain the target code dataset.

[0184] In some implementations, in order to optimize and re-select high-quality candidate datasets based on code learning difficulty to construct a final training set that ensures both quality and diversity of difficulty distribution, clustering can be performed based on the training difficulty score of candidate samples and their semantic category set. Dynamic quota sampling can be implemented based on the cluster size and average difficulty to determine the final target code sample data from multiple candidate code sample data. This combines difficulty assessment with category set diversity control to achieve a fine balance between data quantity, quality and diversity.

[0185] The target code sample data can be a single code sample data unit that is finally selected and retained from the set of candidate code sample data after dynamic quota sampling based on training difficulty score and semantic category set clustering.

[0186] The target code dataset can be the final data set consisting of all target code sample data.

[0187] In some implementations, to construct a dataset that guarantees both quality and breadth and depth, a clustering-based dynamic quota category set deduplication algorithm can be used. Specifically, semantically similar samples can first be grouped into the same category set. Then, based on the size and content difficulty of each category set, the number of samples to retain from that category set is dynamically determined. Finally, within each category set, candidate code sample data with high training difficulty scores are preferentially retained as target code sample data.

[0188] Please refer to Figure 4 Specifically, a lightweight sentence vector model (such as paraphrase-multilingual-MiniLM-L12-v2) can be used to convert the text field (task description information) in each candidate code sample data into a normalized semantic vector. Then, a density-based clustering algorithm (such as HDBSCAN) is used to cluster these vectors. During this process, an appropriate `min_cluster_size` (e.g., 20) can be set to control the clustering granularity. After clustering, each candidate code sample data is assigned a cluster label. Those with a label of -1 are considered noise points (not belonging to any explicit category set), while the remaining candidate code samples with the same label belong to the same semantic category set (cluster).

[0189] Furthermore, for each non-noise cluster (category set), its dynamic retention quota can be calculated. First, obtain the sample size n of the cluster. Then, calculate the average training difficulty score (difficulty) of all samples within the cluster, which serves as the average difficulty value (avg_diff) for the category set. Next, based on the cluster size n and the average difficulty value avg_diff, calculate the number of samples (quota) that should be retained for the cluster using a dynamic quota function. This ensures that larger category sets with higher average difficulty retain more samples to fully represent their diversity and challenge.

[0190] For example, a dynamic quota function can be as follows: ; in, , , For adjustable parameters (e.g.) Set to 1.2, Set to 0.5, (Set to 50), the min function ensures that the quota does not exceed an upper limit gamma. For class sets with particularly high average difficulty, the quota may be close to gamma; for small clusters or simple class sets, the quota will be reduced proportionally.

[0191] Furthermore, within each cluster (category set), the candidate code sample data can be sorted in descending order according to their training difficulty scores. Then, starting from the top of the sorted list, the top quota of samples (i.e., those samples with the highest difficulty in that category set) are selected and retained as part of that category set's contribution to the final target code dataset. The specific process can be as follows: ; The target code sample data from all clusters (category sets) are collected to form the target code dataset.

[0192] For example, suppose clustering yields three categories: ranking algorithms, file operations, and network requests, with sample sizes of 150, 40, and 80 respectively, and average difficulties of 0.6, 0.3, and 0.7 respectively. According to the dynamic quota formula, the ranking algorithm category set may retain approximately 40 samples, the file operation category set approximately 15, and the network request category set approximately 45. Ultimately, approximately 100 target samples are selected from these 270 candidate samples.

[0193] In some implementations, global deduplication based on the MinHash algorithm can be used first to remove similar candidate code sample data before clustering.

[0194] In some implementations, the training difficulty scores of multiple deduplicated candidate code samples can be ranked directly, and the top-k candidate code samples with the highest training difficulty scores can be selected as the target code samples to obtain the target code dataset. The number of k can be determined according to the actual situation, such as 200, 300, etc.

[0195] This application embodiment obtains a preset code dataset, which includes multiple preset code sample data. Each preset code sample data contains multiple code statements and code description information for the multiple code statements. For each preset code sample data, a readability index is calculated based on the length of each code statement, a character density index is calculated based on the proportion of preset type characters in the multiple code statements, and a code complexity index is calculated based on the proportion of preset logical statements in the multiple code statements. A corresponding quality score is calculated by combining the readability index, the character density index, and the code complexity index. Multiple candidate code sample data are selected from the multiple preset code sample data according to the quality score corresponding to each preset code sample data. The semantic similarity between the multiple code statements in each candidate code sample data and the corresponding code description information is determined, and the information entropy of the multiple syntactic structure types contained in each candidate code sample data is determined. A training difficulty score for each candidate code sample data is determined by combining the semantic similarity and the information entropy. Based on the training difficulty score of each candidate code sample data, multiple target code sample data are determined from the multiple candidate code sample data to obtain a target code dataset. Therefore, through multi-dimensional intrinsic code quality assessment and semantic structure analysis, high-quality, semantically consistent, and reasonably distributed target code sample data can be accurately selected from code datasets. Specifically, compared to related technologies that rely on manual rules or simple statistical filtering, this application not only considers the surface features of the code but also deeply evaluates its readability, content density, logical complexity, and description, effectively filtering out low-quality code with poor readability, sparse effective content, or overly simple logic. Simultaneously, by quantifying the complexity and diversity of code structure through semantic similarity and information entropy, it effectively filters out low-quality code samples with small semantic understanding gaps or simple structures, ensuring that the cleaned code data possesses both high quality and training diversity. In summary, this application can improve the quality of code data in code datasets.

[0196] In some implementations, to construct a final training set with a balanced topic distribution and comprehensive difficulty coverage, multiple category sets can be obtained by semantically clustering candidate samples based on task description information. The average difficulty of each category is calculated and the quota is dynamically determined. Then, target samples are selected by sorting by difficulty. This combines topic diversity control with difficulty-priority selection, achieving an optimal balance between data scale, topic coverage, and training value. For example, step 105 may include: (105.1) Based on the task description information of each candidate code sample data, classify multiple candidate code sample data to obtain multiple category sets; (105.2) For each category set, calculate the mean of multiple training difficulty scores corresponding to the multiple candidate code sample data contained therein, and obtain the corresponding average difficulty value; (105.3) Based on the number of samples and the average difficulty value of the multiple second code samples contained in each category set, calculate the number of samples allocated to each category set; (105.4) For each category set, sort the multiple candidate code sample data according to the training difficulty score corresponding to each candidate code sample data to obtain the sorting result, and extract the corresponding multiple target code sample data from the sorting result based on the corresponding sample allocation quantity; (105.5) Based on the multiple target code sample data corresponding to each category set, the target code dataset corresponding to multiple category sets is obtained.

[0197] The task description information can be natural language text used to describe the problem that the code sample is to solve or the function it implements, such as code comments, problem descriptions, or function descriptions, like "implement quicksort with a time complexity of O(nlogn)", which can be used for semantic clustering to classify the topic categories to which the code sample belongs.

[0198] The category set can be an independent group obtained by clustering multiple candidate code sample data based on the semantic similarity of task description information. The samples in each group have high similarity in task topics. For example, all samples involving ranking algorithms are classified into the same category set.

[0199] The average difficulty value can be obtained by calculating the arithmetic mean of the training difficulty scores of all candidate code sample data in a category set. This value reflects the overall learning difficulty level of all candidate code sample data in that category set.

[0200] The number of samples allocated can be the number of target code sample data that should be retained in a certain category set, which is dynamically calculated based on the number of samples (i.e. cluster size) and the average difficulty value of a certain category set according to a preset dynamic quota calculation rule. This can be used to balance the differences in data volume and difficulty between different topics.

[0201] The sorting result can be an ordered sequence obtained by arranging all candidate code sample data within a category set according to their respective training difficulty scores from high to low (or from low to high).

[0202] For example, a lightweight and efficient sentence vector model (e.g., sentence-transformers / paraphrase-multilingual-MiniLM-L12-v2) can be used to convert the task description information of all candidate code sample data into high-dimensional semantic vectors (embeddings). Then, a clustering algorithm suitable for handling unevenly dense data (e.g., HDBSCAN) is used to cluster all these semantic vectors. After the algorithm executes, each candidate code sample is assigned a cluster label. Samples with non-negative label values ​​belong to the same semantic topic, forming a category set; samples with a label of -1 are considered noise or outliers and do not belong to any specific topic. For example, all samples describing "quicksort, mergesort, and heapsort" have semantic vectors that are close to each other in space and are likely to be clustered by HDBSCAN into the same sorting algorithm category set.

[0203] Furthermore, for each non-noise category set, we can obtain all candidate code sample data contained in that set, read their training difficulty scores, and calculate the arithmetic mean of these scores. For example, if category set A has 3 candidate code sample data, then its average difficulty value is the mean of the training difficulty scores of these 3 candidate code sample data. In this way, the inherent learning challenge of different topics can be quantified.

[0204] Furthermore, for each category set, given its number of samples n and average difficulty value, the number of sample assignments corresponding to this category set can be calculated using the following function: ; in, , , For adjustable parameters (e.g.) Set to 1.2, Set to 0.5, (Set to 50), the min function ensures that the quota does not exceed a certain limit. For a set of categories with particularly high average difficulty, the quota may be close to... For small clusters or simple set of categories, the quota will be reduced proportionally.

[0205] Furthermore, within each cluster (category set), the candidate code sample data can be sorted in descending order according to their training difficulty scores to obtain the sorting results. Then, starting from the top of the sorting results, the top quota of samples (i.e., those samples with the highest difficulty in that category set) are selected and retained as the contribution of that category set to the final target code dataset. The specific process can be as follows: ; TOP_K_CLUSTER is the maximum quota limit, which can be set to 30. When the calculated quota exceeds TOP_K_CLUSTER, the final target code sample data retained will be TOP_K_CLUSTER (i.e., a maximum of 30 entries will be retained).

[0206] The target code sample data from all clusters (category sets) are collected to form the target code dataset.

[0207] In this way, difficulty-based screening within the cluster set was completed, ensuring that the final selected samples are not only of high quality, but also the most challenging to learn within their respective topics.

[0208] In some implementations, to further ensure the security and legal compliance of training data and avoid introducing code with security vulnerabilities or legal risks, in addition to selecting high-quality code samples, candidate code can be matched with a pre-defined list of dangerous interfaces to exclude malicious code, and license information extracted from the text using regular expressions can be compared with a list of restrictive licenses to exclude samples with legal risks. This achieves a dual security screening from functional safety to open-source compliance. For example, after step 103, i.e., after "selecting multiple candidate code sample data from multiple pre-defined code sample data according to the quality score corresponding to each pre-defined code sample data", the following may also be included: (A.1) Obtain a preset list of risk interfaces, and match each candidate code sample data with the list of risk interfaces to obtain the first matching result; (A.2) Based on the first matching result, multiple candidate code sample data are filtered to obtain multiple candidate code sample data after filtering; (A.3) Extract the license information of each candidate code sample data after filtering according to the preset regular expression, and match the license information with the preset license restriction list to obtain the second matching result; (A.4) Based on the second matching result, the multiple candidate code sample data after screening are filtered to obtain multiple candidate code sample data after license screening.

[0209] The risk interface list can be a predefined collection of programming interfaces or function names that may pose a threat to system security. For example, in Python, the list may include dangerous function names with high privileges or dynamic execution capabilities, such as exec, eval, and os.system.

[0210] The first matching result can be a judgment on whether the sample contains any of the dangerous interfaces listed in the list, obtained by comparing the code content in the candidate code sample data with each item in the risk interface list.

[0211] The regular expression can be a pre-designed text pattern rule used to identify and extract specific license keywords (such as MIT, GPL, Apache) from the text fields of the code sample.

[0212] The license information can be a string extracted from the text description or comments of the code sample that indicates the type of open source license to which the code is subject, such as MIT License, GPL-3.0, or Apache 2.0.

[0213] The license restriction list can be a predefined set of open-source license identifiers that are considered too restrictive to be included in public training datasets. For example, it may include licenses such as gpl-2.0, gpl-3.0, and agpl-3.0. This list is used to filter out code samples that may impose legal constraints on the subsequent use and distribution of the model.

[0214] The second matching result can be a judgment on whether the license belongs to a restricted type by comparing the extracted license information with the license restriction list, which is used to decide whether the sample should be filtered out based on legal risks.

[0215] For example, a predefined list of risky APIs (RISK_API) can be defined. This list contains the names or patterns of API calls considered to pose security risks; for example, for the Python language, it could include: APIs such as exec, eval, __import__, os.system, subprocess.call, input, and open can be used to execute arbitrary code, perform dangerous system operations, or introduce security vulnerabilities.

[0216] Furthermore, for each candidate code sample, its `code` field can be iterated. Specifically, each item in the risk interface list can be searched within the code string. If any item in the risk interface list is found, the first match marks the sample as "risky"; otherwise, it is marked as "safe". To improve accuracy, a regular expression matching word boundaries (\b) can be constructed to avoid matching variable names or string literals containing these keywords (e.g., avoiding misinterpreting the variable name `execute` as `exec`). In this way, code samples containing potentially dangerous operations can be effectively identified, providing a basis for subsequent security filtering.

[0217] Further, all candidate code sample data with the first matching result of "risk exists" can be directly filtered out and not retained. After that, the remaining samples after screening constitute multiple candidate code sample data after screening, and these samples are considered relatively safe at the API call level.

[0218] Further, the copyright license information of the candidate code sample data after screening can be checked to avoid introducing license codes with strong infectivity or overly restrictive licenses, resulting in legal risks. Specifically, a license restriction list can be preset, including specific license versions that are more restrictive and may not be suitable for inclusion in the publicly available training dataset, such as {gpl-2.0, gpl-3.0, agpl-3.0}. Then, for each candidate code sample data after screening, the license information is extracted from its text and / or description fields. Specifically, it can be implemented through a preset regular expression, which is used to match the license name string following leading words such as "license:", "许可证:".

[0219] Further, after the original license string is extracted, it can be matched with the preset license restriction list to obtain the second matching result. If the second matching result indicates that it hits the license restriction list, the second matching result is marked as "restrictive license", and the candidate code sample data with the second matching result of "restrictive license" (i.e., hitting the license restriction list) is filtered out; if it is other open source licenses or no clear license is detected, it is marked as "other". In this way, the license type declared by the code sample can be automatically identified, and those samples with strong infectivity clauses can be located.

[0220] In some embodiments, in order to automatically verify the accuracy and relevance of the internal text description of the code sample to ensure the semantic consistency of the training data, the code and its description can be encoded into high-dimensional semantic vectors respectively through a pre-trained language model, and the similarity between the two is calculated as the consistency score. Then, samples with descriptions that seriously do not match the code are filtered out through threshold comparison, so as to filter out low-quality or misleading data from the semantic level and improve the internal logical consistency of the dataset. Exemplarily, before step 104, that is, before "determining the code semantic similarity between multiple code statements in each candidate code sample data and the corresponding code description information", it can also include: (B.1) For each candidate code sample data, obtain the first semantic vector corresponding to multiple code statements and the second semantic vector corresponding to the code description information; (B.2) Based on the similarity between the first semantic vector and the second semantic vector, calculate the code semantic consistency score between multiple code statements and code description information; (B.3) Compare the code semantic consistency score with the preset consistency threshold to obtain the comparison result; (B.4) Based on the comparison results of each candidate code sample data, multiple candidate code sample data are filtered to obtain multiple candidate code sample data after semantic filtering.

[0221] The first semantic vector can be a dense vector representation of code statements in candidate code sample data, which is transformed by a pre-trained multilingual sentence vector model (such as the SentenceTransformer model). This vector contains the semantic information of the code and can be used to calculate semantic similarity.

[0222] The second semantic vector can be a dense vector representation of fixed dimensions, which is generated by converting the code description information in the candidate code sample data into a single pre-trained multilingual sentence vector model. This vector contains the semantic information describing the text.

[0223] Among them, the code semantic consistency score can be a numerical value used to quantify the degree of semantic matching between code statements and code description information. It can be obtained by calculating the dot product or cosine similarity between the first semantic vector and the second semantic vector. The higher the score, the better the semantic consistency between the description and the code.

[0224] The comparison result can be a logical judgment conclusion obtained by comparing the calculated code semantic consistency score with a preset consistency threshold (e.g., 0.35), such as a Boolean value, which directly indicates whether the sample should be filtered out due to semantic inconsistency.

[0225] For example, a pre-trained, lightweight multilingual sentence vector model can be selected to obtain the first and second semantic vectors. For instance, multiple code statements (i.e., the `code` field) can be input into a pre-trained sentence-transformers model or a paraphrase-multilingual-MiniLM-L12-v2 model. Taking the paraphrase-multilingual-MiniLM-L12-v2 model as an example, multiple code statements can be input as text to the model to obtain the first semantic vectors corresponding to the multiple code statements, and the code description information can be input into the same model to obtain the corresponding second semantic vectors.

[0226] Furthermore, the similarity between the first and second semantic vectors can be calculated by calculating cosine similarity. The specific calculation process is as follows: ; in, Represents the first semantic vector. Represents the second semantic vector The transpose of . This similarity can be used as a score for the semantic consistency between multiple code statements and code description information.

[0227] For example, a consistency threshold can be set to 0.35, or other values ​​can be set according to the actual situation. The semantic consistency score of the candidate code sample data is then compared with the consistency threshold to obtain the comparison result. Afterwards, based on the comparison result of each candidate code sample data, the candidate code sample data is filtered. For example, if the semantic consistency score of a candidate code sample data is 0.12, which is less than 0.35, it is discarded; if the semantic consistency score of a candidate code sample data is 0.82, which is greater than 0.35, it is retained. This directly filters out low-quality samples whose descriptions severely mismatch the code, improving the efficiency of the entire cleaning process and the inherent consistency quality of the final dataset.

[0228] Please refer to Figure 5 In some implementations, combined with Figure 5 The overall process of the embodiments of this application is described below. For example, the code data processing method may begin by loading a preset code dataset and first performing basic text cleaning, such as removing HTML tags and URL links from the samples, removing leading and trailing whitespace characters, and filtering out samples with too few characters (e.g., less than 100 characters) to ensure basic integrity. Subsequently, language filtering is performed, retaining only code samples containing both Chinese and English, laying the foundation for subsequent targeted quality assessment.

[0229] Further, the process enters the core quality assessment and screening stage. First, code quality filtering is performed based on readability, complexity, and completeness. This involves calculating readability metrics, character density metrics, and code complexity metrics and combining them into a quality score. This step initially filters out well-structured and logically clear candidate code samples. Next, security and license filtering are performed sequentially. By matching a list of risky interfaces and extracting and comparing license information, samples with security vulnerabilities or legal compliance risks are eliminated to ensure data security. Then, a sentence vector model is used to calculate the semantic similarity between the code and its corresponding description, implementing description-code alignment filtering to retain high-quality samples with consistent semantics.

[0230] Building upon this foundation, the method further refines the evaluation of sample difficulty. By integrating syntactic structure analysis based on AST (Abstract Syntax Tree) parsing with semantic analysis based on sentence vector models, a comprehensive semantic-syntactic difficulty calculation is performed to obtain a training difficulty score for each sample. Subsequently, the MinHash algorithm is used for global deduplication, eliminating highly similar duplicate samples in the dataset. Finally, a dynamic quota-based topic deduplication algorithm based on clustering is employed. Semantic clustering is performed based on the task description of the samples, and quotas are dynamically allocated according to the number of samples and average difficulty in each category. The most representative and appropriately difficult samples are selected from each category, and these filtered samples are ultimately retained to form a high-quality, diverse, and reasonably distributed target code dataset. The entire process is interconnected, achieving automated, multi-dimensional cleaning and construction from raw code data to a high-quality training dataset.

[0231] Please see Figure 6 This application also provides a code data processing apparatus that can implement the above-described code data processing method. The code data processing apparatus includes: The acquisition module 61 is used to acquire a preset code dataset. The preset code dataset includes multiple preset code sample data. Each preset code sample data contains multiple code statements and code description information for the multiple code statements. The calculation module 62 is used to calculate the readability index for each preset code sample data, based on the length of each code statement, to calculate the character density index based on the proportion of preset type characters in multiple code statements, and to calculate the code complexity index based on the proportion of preset logical statements in multiple code statements. The corresponding quality score is calculated by combining the readability index, character density index and code complexity index. The filtering module 63 is used to filter out multiple candidate code sample data from multiple preset code sample data according to the quality score corresponding to each preset code sample data; The first determining module 64 is used to determine the code semantic similarity between multiple code statements in each candidate code sample data and the corresponding code description information, and to determine the information entropy of multiple syntactic structure types contained in each candidate code sample data. Combining the code semantic similarity and information entropy of each candidate code sample data, the module determines the training difficulty score of each candidate code sample data. The second determination module 65 is used to determine multiple target code sample data from multiple candidate code sample data based on the training difficulty score of each candidate code sample data, thereby obtaining the target code dataset.

[0232] The specific implementation of this code data processing device is basically the same as the specific embodiment of the code data processing method described above, and will not be repeated here. Subject to meeting the requirements of the embodiments of this application, the code data processing device may also be equipped with other functional modules to implement the code data processing method described above.

[0233] This application also provides a computer device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described code data processing method. This computer device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0234] Please see Figure 7 , Figure 7 The hardware structure of a computer device according to another embodiment is illustrated. The computer device includes: The processor 71 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application. The memory 72 can be implemented as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 72 can store the operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 72 and is called and executed by the processor 71 using the code data processing method of the embodiments of this application. Input / output interface 73 is used to implement information input and output; The communication interface 74 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.). Bus 75 transmits information between various components of the device (e.g., processor 71, memory 72, input / output interface 73, and communication interface 74); The processor 71, memory 72, input / output interface 73, and communication interface 74 are connected to each other within the device via bus 75.

[0235] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described code data processing method.

[0236] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0237] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0238] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0239] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0240] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0241] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0242] It should be understood that in this application, "at least one" and "several" refer to one or more, and "multiple" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0243] In the embodiments provided in this application, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0244] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0245] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0246] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0247] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A code data processing method, characterized in that, The method includes: Obtain a preset code dataset, which includes multiple preset code sample data, each preset code sample data containing multiple code statements and code description information for the multiple code statements; For each preset code sample data, a readability index is calculated based on the length of each code statement, a character density index is calculated based on the proportion of preset type characters in the multiple code statements, and a code complexity index is calculated based on the proportion of preset logical statements in the multiple code statements. The corresponding quality score is then calculated by combining the readability index, the character density index, and the code complexity index. Based on the quality score corresponding to each preset code sample data, multiple candidate code sample data are selected from the multiple preset code sample data; Determine the semantic similarity between the multiple code statements in each candidate code sample data and the corresponding code description information, and determine the information entropy of the multiple syntactic structure types contained in each candidate code sample data. Combine the semantic similarity and the information entropy of each candidate code sample data to determine the training difficulty score of each candidate code sample data. Based on the training difficulty score of each candidate code sample data, multiple target code sample data are determined from the multiple candidate code sample data to obtain the target code dataset.

2. The code data processing method according to claim 1, characterized in that, The calculation of the readability metric based on the length of each code statement includes: Obtain the first character count for each code statement, and identify code statements whose first character count is greater than a preset character count threshold as target code statements; The number of first statements and the number of second statements containing the target code statements in the preset code sample data are determined, and the readability index of the preset code sample data is obtained based on the ratio between the number of first statements and the number of second statements.

3. The code data processing method according to claim 1, characterized in that, The preset character types include alphanumeric characters and numeric characters. The calculation of the character density index based on the proportion of preset character types in the multiple code statements includes: Obtain the total number of characters in the preset code sample data, and the number of second characters corresponding to the letter type characters and the number type characters contained in the preset code sample data; The larger value between the total number of characters and the preset first reference value is selected as the target total number of characters; The character density index of the preset code sample data is calculated based on the ratio between the second character count and the target total character count.

4. The code data processing method according to claim 1, characterized in that, The step of calculating the code complexity index based on the proportion of preset logical statements among the multiple code statements includes: Obtain the number of second statements in the plurality of code statements contained in the preset code sample data, and the number of third statements in the preset logical statements contained therein; Calculate the initial code complexity index based on the ratio between the number of the third statement and the number of the second statement; The larger value between the initial code complexity index and the preset second reference value is selected as the code complexity index of the preset code sample data.

5. The code data processing method according to claim 1, characterized in that, The process of calculating the corresponding quality score by combining the readability metric, the character density metric, and the code complexity metric includes: Obtain the first weight corresponding to the readability index and the second weight corresponding to the character density index; Based on the first weight, the readability index is adjusted to obtain the target readability index; Based on the relationship between the character density index and the preset character density threshold, a density identifier value is determined, and the density identifier value is adjusted based on the second weight to obtain the target character density index. Based on the ratio between the code complexity index and the preset benchmark value, an adjustment index value is determined, and the smaller value between the preset third reference value and the adjustment index value is selected as the target code complexity index. Based on a preset first reference value, the difference between the target readability index, the target character density index, and the target code complexity index is calculated sequentially to obtain the quality score corresponding to the preset code sample data.

6. The code data processing method according to claim 1, characterized in that, The determination of the information entropy of multiple syntactic structure types contained in each candidate code sample data includes: Obtain the multiple syntax structure types contained in each candidate code sample data, and the frequency of occurrence of each syntax structure type in the multiple code statements; The total frequency of the target occurrence is obtained by summing the occurrence frequencies corresponding to the multiple syntactic structure types. For each syntax structure type, the corresponding type distribution frequency is calculated based on the ratio between the corresponding occurrence frequency and the total occurrence frequency of the target. Based on the frequency distribution of multiple types corresponding to the multiple syntax structure types, the information entropy of the multiple syntax structure types contained in each candidate code sample data is calculated.

7. The code data processing method according to claim 1, characterized in that, The step of combining the semantic similarity of the code and the information entropy of each candidate code sample data to determine the training difficulty score of each candidate code sample data includes: For each candidate code sample data, the corresponding semantic understanding difficulty is obtained based on the difference between the preset first reference value and the semantic similarity of the code; Obtain preset third and fourth weights, and adjust the semantic understanding difficulty based on the third weight to obtain the target semantic understanding difficulty; and adjust the information entropy based on the fourth weight to obtain the target information entropy. The training difficulty score for each candidate code sample data is obtained based on the sum of the target semantic understanding difficulty and the target information entropy.

8. The code data processing method according to claim 1, characterized in that, Each preset code sample data also includes task description information corresponding to the multiple code statements. Based on the training difficulty score of each candidate code sample data, multiple target code sample data are determined from the multiple candidate code sample data to obtain a target code dataset, including: Based on the task description information of each candidate code sample data, the multiple candidate code sample data are classified to obtain multiple category sets; For each category set, the mean of multiple training difficulty scores corresponding to the multiple candidate code sample data contained therein is calculated to obtain the corresponding average difficulty value; Based on the number of samples of multiple second code samples contained in each category set and the average difficulty value, calculate the number of samples allocated to each category set; For each category set, multiple candidate code sample data are sorted according to the training difficulty score corresponding to each candidate code sample data to obtain a sorting result, and multiple target code sample data are extracted from the sorting result based on the corresponding sample allocation quantity; Based on the target code sample data corresponding to each category set, the target code dataset corresponding to the multiple category sets is obtained.

9. The code data processing method according to claim 1, characterized in that, After selecting multiple candidate code sample data from the multiple preset code sample data according to the quality score corresponding to each preset code sample data, the method further includes: Obtain a preset list of risk interfaces, and match each candidate code sample data with the list of risk interfaces to obtain a first matching result; Based on the first matching result, the multiple candidate code sample data are filtered to obtain multiple filtered candidate code sample data; The license information of each candidate code sample data after filtering is extracted according to a preset regular expression, and the license information is matched with a preset license restriction list to obtain a second matching result; Based on the second matching result, the multiple candidate code sample data after filtering are further filtered to obtain multiple candidate code sample data after license filtering.

10. The code data processing method according to claim 1, characterized in that, Before determining the code semantic similarity between the plurality of code statements in each candidate code sample data and the corresponding code description information, the method further includes: For each candidate code sample data, obtain the first semantic vector corresponding to the plurality of code statements and the second semantic vector corresponding to the code description information; Based on the similarity between the first semantic vector and the second semantic vector, a code semantic consistency score is calculated between the plurality of code statements and the code description information; The code semantic consistency score is compared with a preset consistency threshold to obtain the comparison result; Based on the comparison results of each candidate code sample data, the multiple candidate code sample data are filtered to obtain multiple candidate code sample data after semantic filtering.

11. A code data processing device, characterized in that, The device includes: The acquisition module is used to acquire a preset code dataset, which includes multiple preset code sample data, each preset code sample data containing multiple code statements and code description information for the multiple code statements; The calculation module is used to calculate a readability index for each preset code sample data, based on the length of each code statement, to calculate a character density index based on the proportion of preset type characters in the multiple code statements, and to calculate a code complexity index based on the proportion of preset logical statements in the multiple code statements. The module then combines the readability index, the character density index, and the code complexity index to calculate the corresponding quality score. The filtering module is used to filter out multiple candidate code sample data from the multiple preset code sample data according to the quality score corresponding to each preset code sample data; The first determining module is used to determine the code semantic similarity between the plurality of code statements in each candidate code sample data and the corresponding code description information, and to determine the information entropy of the plurality of syntactic structure types contained in each candidate code sample data. Combining the code semantic similarity and the information entropy of each candidate code sample data, the module determines the training difficulty score of each candidate code sample data. The second determining module is used to determine multiple target code sample data from the multiple candidate code sample data based on the training difficulty score of each candidate code sample data, thereby obtaining a target code dataset.

12. A computer device, characterized in that, The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the code data processing method according to any one of claims 1 to 10.

13. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the code data processing method according to any one of claims 1 to 10.

Images