Source code security detection method based on semantic features and lightweight model

Abstract

The application belongs to the technical field of safety detection, and discloses a source code safety detection method based on semantic features and a lightweight model; the method comprises the following steps: acquiring multi-source data of a to-be-detected project in a version iteration process; extracting demand semantic features and code semantic features, performing semantic splitting, and outputting a code structure binary tuple; identifying a business scenario category, extracting a safety calling feature, and simultaneously performing interference elimination; positioning a risk turning point, identifying a pseudo-sudden increase turning point, and constructing a real risk change path; extracting a stage risk feature, performing dimension reduction compression, inputting a low-dimensional risk feature vector into a preset lightweight risk prediction model to obtain a safety risk score, and simultaneously generating a priority repair suggestion list in combination with a corresponding risk feature set; and the positioning accuracy, risk identification accuracy, and detection efficiency in a version iteration scenario of source code safety detection are improved.

Description

Technical Field

[0001] This application relates to the field of security detection technology, and more specifically, to a source code security detection method based on semantic features and a lightweight model. Background Technology

[0002] Source code security testing during the version iteration process of software projects is a key link and core step in the field of software engineering. Continuous risk monitoring and vulnerability analysis of this process are used to ensure the security of the overall system operation, as well as to ensure the quality of software code and the development efficiency of version iteration. However, traditional security testing methods still face many technical bottlenecks when facing more complex version iteration scenarios.

[0003] In version iteration scenarios, a single code commit typically includes both adjustments to business functionality and exception handling. For example, a code snippet might contain core data query logic as well as boundary validation processes such as input null checks or exception handling. However, traditional security detection methods generally scan the source file directly, ignoring the significant semantic differences between core logic code and boundary handling code. This allows core functional business risks to be interfered with by a large amount of edge-based defense code. Furthermore, calls to third-party libraries in projects often involve complex contexts. However, traditional security detection methods do not match the call characteristics of third-party libraries with the protection coverage of boundary handling code, failing to distinguish between constrained calls and genuine high-risk calls. This can easily lead to misjudging protected calls as high-risk vulnerabilities, triggering numerous false security alerts. Simultaneously, in continuous... On the version iteration timeline, drastic fluctuations in risk indicators do not necessarily represent the introduction of new risks. Partial code refactoring or file migration can cause the original risk characteristics to redistribute in a short period of time, resulting in a numerical jump. However, traditional security detection methods lack the ability to identify and eliminate noise interference during risk changes, which can easily lead to misjudging pseudo-surges caused by refactoring as real risks. On the other hand, source code security detection is a process that requires high-frequency execution and extremely high feedback timeliness. Directly inputting all code features into the model would cause a surge in resource overhead. Traditional security detection methods generally use prediction models that map all features, which lack dimensionality reduction and compression based on real risk changes. This results in an excessive inference burden on the prediction model and difficulty in providing rapid feedback with each code commit, seriously affecting the execution efficiency of source code security detection.

[0004] In view of this, this application proposes a source code security detection method based on semantic features and a lightweight model to solve the above problems. Summary of the Invention

[0005] To overcome the aforementioned shortcomings of existing technologies and achieve the above objectives, this application provides the following technical solution: a source code security detection method based on semantic features and a lightweight model, comprising: S1. Obtain the code commit records, third-party library call records, and corresponding requirement change documents and code review logs of the project to be tested during the version iteration process; S2. Extract semantic features of requirements based on requirement change documents and semantic features of code based on code commit records; identify the mapping relationship between requirement semantic features and code semantic features; perform semantic decomposition on code commit records based on the mapping relationship; and output a code structure tuple consisting of core logic code and boundary handling code. S3. Identify business scenario categories for the core logic code and classify them according to these categories to obtain business scenario groups; extract security call features from third-party library call records in each business scenario group; combine the corresponding boundary processing code to perform interference removal on the security call features to obtain the risk feature set of the corresponding business scenario group; S4. Normalize the risk feature set for each business scenario group and output a security risk indicator sequence; perform time period fitting on the security risk indicator sequence to locate risk inflection points; identify the risk feature distribution state before and after the risk inflection point, and identify pseudo-increase inflection points by combining code review logs; remove pseudo-increase inflection points and construct the real risk change path based on the remaining risk inflection points. S5. Extract stage risk features based on the actual risk change path, reduce the dimensionality of the stage risk features to output a low-dimensional risk feature vector, input the low-dimensional risk feature vector into a preset lightweight risk prediction model to obtain a safety risk score, and generate a priority repair suggestion list by combining the corresponding risk feature set.

[0006] Furthermore, methods for identifying the mapping relationship between requirement semantic features and code semantic features include: The requirement change document is segmented into words to extract the set of action words and the set of business nouns. The set of action words and the set of business nouns are then transformed into requirement feature vectors as requirement semantic features. Obtain the abstract syntax tree corresponding to the code commit record, identify the changed strings and instructions, and combine them into a code feature vector as the code semantic features; Calculate the cosine similarity between the requirement feature vector and each code feature vector, and use this cosine similarity as the mapping relationship between the corresponding requirement semantic features and code semantic features.

[0007] Furthermore, methods for semantically splitting code commit records include: Set core judgment thresholds and boundary judgment thresholds, and ensure that the core judgment threshold is greater than the boundary judgment threshold; When the cosine similarity of the mapping relationship is higher than the core judgment threshold, the corresponding code submission record is used as the core logic code; if the cosine similarity is lower than the boundary judgment threshold, and the corresponding code submission record is used to handle abnormal situations, the code submission record is used as the boundary handling code. When the cosine similarity corresponding to the mapping relationship is between the core judgment threshold and the boundary judgment threshold, the corresponding code submission record is divided into code statements to form a code statement sequence; the code feature vector of each code statement in the code statement sequence is constructed, the cosine similarity between each code feature vector and the corresponding requirement feature vector is calculated, and threshold judgment is performed one by one; All core logic code and boundary handling code are combined into code structure tuples.

[0008] Furthermore, the core logic code is categorized into business scenario types, and then classified according to these categories to obtain business scenario groups. Within each business scenario group, the methods for extracting security call features from third-party library call records include: Extract the set of operation keywords from the core logic code, identify the number of characters in the intersection of the set of operation keywords and the set of template keywords for each preset scenario, and calculate the ratio of the number of characters in the intersection to the total number of characters in the set of operation keywords to obtain the template matching degree; The scenario category corresponding to the scenario with the highest template matching degree is taken as the business scenario category, and all core logic code belonging to the same business scenario category is aggregated to obtain business scenario grouping; For the core logic code in each business scenario group, identify the call location recorded in the third-party library call record; extract the call function name and parameter passing path at the call location, and combine them into a secure call feature.

[0009] Furthermore, methods for performing interference removal on secure call characteristics include: For each security call feature, obtain the set of security verification patterns that match the security call feature; perform pattern matching between the boundary processing code corresponding to the security call feature and the corresponding set of security verification patterns, and calculate the verification coverage of the boundary processing code to the corresponding set of security verification patterns; When the verification coverage is higher than or equal to the preset integrity threshold, it is determined that the security call feature is effectively protected, and the corresponding security call feature is removed as an interference item. When the verification coverage is lower than the preset integrity threshold, the security call features are retained; the security call features retained in all business scenario groups are integrated to obtain the risk feature set of the corresponding business scenario group.

[0010] Furthermore, the risk feature set for each business scenario group is normalized to output a security risk indicator sequence; time period fitting is performed on the security risk indicator sequence to locate risk inflection points, including: Obtain the preset risk level corresponding to each risk feature in the risk feature set; at the same time, identify the submission timestamp of each core logic code in the business scenario group corresponding to the risk feature set based on the code submission record, and align the risk features in the same business scenario group according to their respective submission timestamps; At each submission timestamp, calculate the sum of risk levels of all risk features in each business scenario group, and calculate the ratio of the sum of risk levels to the number of risk features in the business scenario group to obtain the normalized risk value of the corresponding business scenario group. The normalized risk values ​​under each submission timestamp are arranged according to the order of submission timestamps to obtain a sequence of security risk indicators. A sliding time window is constructed to divide the sequence of safety risk indicators, and the risk mean of the normalized risk value in each sliding time window is calculated; Calculate the difference between the average risk values ​​of adjacent sliding time windows. If the difference is greater than a preset turning point threshold, then the boundary time point between adjacent sliding time windows is taken as the risk turning point.

[0011] Furthermore, methods for identifying pseudo-incremental inflection points include: Extract the code review logs of the sliding time window where the risk inflection point is located, and identify whether there are review semantics related to code refactoring or code migration in the code review logs; Obtain the first risk feature set within the preceding sliding time window and the second risk feature set within the following sliding time window for each risk inflection point; identify the intersection risk features between the first and second risk feature sets, and calculate the risk feature overlap rate by the ratio of the number of intersection risk features to the total number of risk features in the second risk feature set. When there are review semantics related to code refactoring or code migration, and the overlap rate of risk features is higher than the preset overlap threshold, the corresponding risk inflection point is determined to be a pseudo-sudden increase inflection point.

[0012] Furthermore, methods for constructing real risk change paths include: Replace the normalized risk value at the pseudo-increase inflection point with the mean of the normalized risk value of the previous sliding time window; at the same time, retain other risk inflection points that were not identified as pseudo-increase inflection points as true risk inflection points. By using adjacent real risk inflection points as dividing boundaries, the complete timeline is divided into multiple continuous risk stages, which are then spliced ​​together in chronological order to obtain the real risk change path.

[0013] Furthermore, methods for extracting stage risk features based on the actual risk change path and then reducing and compressing these stage risk features to output low-dimensional risk feature vectors include: The mean risk of a stage is obtained by calculating the mean of the normalized risk values ​​of all sampling points in each continuous risk stage in the actual risk change path. The maximum number of affected codes in each continuous risk phase is identified as the peak number, and the average risk of the corresponding continuous risk phase and the peak number are combined to form the basic characteristics of that continuous risk phase. Calculate the rate of change of the average risk of adjacent consecutive risk stages and the difference of the peak values ​​of the stages, and combine them into stage correlation features; The basic features and stage-related features of each continuous risk stage are concatenated to obtain the stage risk features; feature items whose rate of change of the stage risk mean is lower than the preset rate of change threshold are deleted from the stage risk features, and the adjusted stage risk features are transformed into a low-dimensional risk feature vector.

[0014] Furthermore, the methods for generating a priority repair suggestion list include: Obtain the preset risk level and the average stage risk of the continuous risk stage to which each risk feature belongs in the corresponding risk feature set; use preset priority weights to perform a weighted summation of the safety risk score, preset risk level and the average stage risk of the continuous risk stage to which it belongs, to obtain the repair priority score of each risk feature; sort all risk features based on the size of the repair priority score to obtain a priority repair suggestion list.

[0015] The technical effects and advantages of the source code security detection method based on semantic features and lightweight models in this application are as follows: By collecting multi-source data required for source code security testing, a source code security testing method based on semantic features and a lightweight model was implemented. Compared with existing experience, semantic decomposition eliminated the detection target positioning deviation caused by the mixing of core logic code and boundary processing code. By performing interference elimination, it distinguished between calls that appear to have risks but have been effectively protected and calls that actually expose risks, avoiding misjudgment problems in third-party library call scenarios. By identifying pseudo-sudden increase inflection points, it filtered out false risk leaps. By performing dimensionality reduction and compression on features, the pre-set lightweight risk prediction model can complete inference and output with low computational overhead, avoiding the inference burden and feedback lag caused by directly inputting high-dimensional features into the model. Overall, it improved the positioning accuracy, risk identification accuracy, and detection efficiency in version iteration scenarios of source code security testing. Attached Figure Description

[0016] Figure 1 This is a schematic diagram of the source code security detection method based on semantic features and a lightweight model proposed in this application. Detailed Implementation

[0017] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0018] Example 1 Please see Figure 1 As shown in this embodiment, the source code security detection method based on semantic features and a lightweight model includes: S1. Obtain the code commit records, third-party library call records, and corresponding requirement change documents and code review logs of the project to be tested during the version iteration process; S2. Extract semantic features of requirements based on requirement change documents and semantic features of code based on code commit records; identify the mapping relationship between requirement semantic features and code semantic features; perform semantic decomposition on code commit records based on the mapping relationship; and output a code structure tuple consisting of core logic code and boundary handling code. S3. Identify business scenario categories for the core logic code and classify them according to these categories to obtain business scenario groups; extract security call features from third-party library call records in each business scenario group; combine the corresponding boundary processing code to perform interference removal on the security call features to obtain the risk feature set of the corresponding business scenario group; S4. Normalize the risk feature set for each business scenario group and output a security risk indicator sequence; perform time period fitting on the security risk indicator sequence to locate risk inflection points; identify the risk feature distribution state before and after the risk inflection point, and identify pseudo-increase inflection points by combining code review logs; remove pseudo-increase inflection points and construct the real risk change path based on the remaining risk inflection points. S5. Extract stage risk features based on the actual risk change path, reduce the dimensionality of the stage risk features to output a low-dimensional risk feature vector, input the low-dimensional risk feature vector into a preset lightweight risk prediction model to obtain a safety risk score, and generate a priority repair suggestion list by combining the corresponding risk feature set.

[0019] In this embodiment, code commit records refer to the records generated by each code commit during the version iteration process of the project under test, including information such as commit timestamps, changed file paths, changed code snippets, and commit descriptions, which are used to reflect the source code modification content and timing in different versions of the project; third-party library call records refer to the call information records of external dependent libraries in the project source code, including information such as the name of the called library, version number, and called function, which are used to reflect the usage of external interfaces related to security risks in the source code; requirement change documents refer to the requirement description information, function adjustment instructions, and change scope restrictions corresponding to the project during version iteration, which are used to reflect the goals and scope of business function changes in version iteration; code review logs refer to the review comments, modification suggestions, review conclusions, and review timestamps generated from the code commit records, which are used to reflect whether the code modifications involve information such as refactoring, migration, exception modification, or other structural corrections.

[0020] Methods for identifying the mapping relationship between requirement semantic features and code semantic features include: The requirement change document is segmented into words to extract the set of action words and the set of business terms. The set of action words and the set of business terms are then transformed into requirement feature vectors as the semantic features of the requirements.

[0021] In this embodiment, the requirement change document is text data. Therefore, by performing word segmentation, the continuous text content is divided into several words with independent semantics. The set of action words that represent functional actions or processing behaviors, such as adding, deleting, updating, and uploading, and the set of nouns that represent business objects, data objects, or resource objects, such as users, orders, paths, and sessions, are identified. The set of action words and the set of business nouns are sorted according to the text order in the original requirement change document. In this embodiment, a pre-trained BERT language model is used to map the text content into vector form, namely the requirement feature vector.

[0022] Obtain the abstract syntax tree corresponding to the code commit record, identify the changed strings and instructions, and combine them into a code feature vector as the code semantic features.

[0023] In this embodiment, the Abstract Syntax Tree (AST) refers to the tree structure representation obtained by parsing the source code content that has changed in the code commit record. Each node is used to represent a syntactic unit or program component unit in the source code, such as a function definition node, a variable definition node, or a condition judgment node.

[0024] The changed strings refer to name-type identifiers added, deleted, or modified in the code, such as function name strings, variable name strings, and interface name strings; the changed instructions refer to program execution actions added or modified in the code commit, such as call instructions, conditional instructions, and assignment instructions; by combining the changed strings and instructions according to the encoding order of code belonging to different modules or different files in the code commit record in the abstract syntax tree, code feature vectors corresponding to different modules and files are generated.

[0025] Calculate the cosine similarity between the requirement feature vector and each code feature vector, and use this cosine similarity as the mapping relationship between the corresponding requirement semantic features and code semantic features.

[0026] The formula for calculating cosine similarity is as follows: ;in, Represents the demand feature vector and a certain code feature vector The cosine similarity is used to calculate the degree of consistency between the semantic features of the requirement and the semantic features of the corresponding code in terms of direction. If the cosine similarity is high, it means that the actions and objects in the corresponding code commit record are closer to the business intent in the requirement change document, and vice versa. The cosine similarity is used as a mapping relationship so that each code commit record can obtain a quantifiable correlation index.

[0027] The methods for semantically splitting code commit records include: Set a core judgment threshold and a boundary judgment threshold, and ensure that the core judgment threshold is greater than the boundary judgment threshold.

[0028] Since there are significant differences in the semantic similarity between different code commit records and requirement change documents, a single threshold cannot be used for judgment. Therefore, based on a large amount of historical semantic judgment experience, two thresholds are set, and the core judgment threshold is made larger than the boundary judgment threshold, forming a three-segment judgment interval.

[0029] The core judgment threshold is used to limit code commit records that are highly consistent with the semantics of the requirement change and can be regarded as the main functional change content; the boundary judgment threshold is used to limit code commit records that are less related to the semantics of the requirement change, generally including code commit records that perform input validation or play a defensive role.

[0030] When the cosine similarity of the mapping relationship is higher than the core judgment threshold, the corresponding code submission record will be used as the core logic code.

[0031] If the cosine similarity is higher than the core judgment threshold, it means that the corresponding code commit record is highly consistent with the goal of the requirement change document at the functional action and business object level. Such code generally directly participates in the core level data processing, so the corresponding code commit record is regarded as core logic code.

[0032] If the cosine similarity is lower than the boundary judgment threshold, and the corresponding code submission record is used to handle abnormal situations, then the code submission record is used as the boundary handling code.

[0033] If the cosine similarity is lower than the boundary judgment threshold, it indicates that the corresponding code commit record has a weak correlation with the core business actions and business objects in the requirement change document. On the other hand, "used to handle abnormal situations" means that the code commit record also includes processing tasks such as input nulling, parameter validity checking, resource release, and alarm prompts to ensure the overall program's stability and security, and does not directly involve business functions.

[0034] If both conditions are met simultaneously—cosine similarity below the corresponding threshold and code commit records used to handle exceptional situations—then the corresponding code commit records can be regarded as boundary processing code. This code can be used to constrain and protect core business processing logic or calls to third-party libraries, thus avoiding misjudging business support code that is semantically weakly related to the requirements but equally important as boundary processing code.

[0035] When the cosine similarity corresponding to the mapping relationship is between the core judgment threshold and the boundary judgment threshold, the corresponding code submission record is divided into code statements to form a code statement sequence.

[0036] When the cosine similarity is between the core judgment threshold and the boundary judgment threshold, it indicates that the corresponding code submission record does not fully conform to the high relevance of the core logic code nor the low relevance of the boundary processing code in terms of overall semantics. This indicates that the code contains a mixture of some business processing code and exception handling code. In this embodiment, the separator between each line of code is used as an identifier to divide all the code in the corresponding code submission record into statements, and they are arranged into a code statement sequence according to the order of each line of code in the source code.

[0037] Construct the code feature vector for each code statement in the code statement sequence, calculate the cosine similarity between each code feature vector and the corresponding requirement feature vector, and perform threshold judgment for each one.

[0038] In this embodiment, an abstract syntax tree is constructed based on the code syntax of each code statement. Then, each code statement is converted into a code feature vector. Each code feature vector is compared with the core judgment threshold and the boundary judgment threshold to classify it into core logic code or boundary processing code.

[0039] If, after evaluating each code statement, there are still code statements falling between the two thresholds, then that code statement is traced upwards according to the abstract syntax tree composed of all code statements in its code block or file, and is assigned to the code type corresponding to the parent node statement.

[0040] All core logic code and boundary handling code are combined into code structure tuples.

[0041] After threshold judgment, two types of code are output: core logic code and boundary processing code. Several core logic codes and boundary processing codes belonging to the same code submission record are combined to form a code structure tuple.

[0042] The core logic code is categorized into business scenario types, and then classified according to these categories to obtain business scenario groups. Within each business scenario group, the methods for extracting security call features from third-party library call records include: Extract the set of operation keywords from the core logic code, identify the number of characters in the intersection of the operation keyword set and the set of template keywords for each preset scenario, and calculate the ratio of the number of characters in the intersection to the total number of characters in the operation keyword set to obtain the template matching degree.

[0043] Operation keywords refer to keywords extracted from the core logic code that can represent the main processing actions and business intentions of the corresponding code content, including code keywords that reflect program functional behavior such as reading, writing, downloading, and uploading; each preset scenario template keyword set refers to a set of keywords corresponding to different business scenarios based on a large amount of historical experience. Business scenarios refer to typical business processing scenarios such as database access scenarios, file operation scenarios, and network communication scenarios.

[0044] The template matching degree is obtained by identifying the number of characters in the intersection of the set of operation keywords and the set of scene template keywords for the corresponding scene, and calculating the ratio of the number of characters in the intersection to the total number of characters in the set of operation keywords. This value is used to indicate the proportion of keywords that match the set of template keywords for a certain scene.

[0045] The scenario category corresponding to the scenario with the highest template matching degree is taken as the business scenario category, and all core logic code belonging to the same business scenario category is aggregated to obtain business scenario grouping.

[0046] After the set of operation keywords and multiple preset scenario template keyword sets have been calculated for template matching degree, the scenario category of the business scenario with the highest template matching degree is selected as the business scenario category of the corresponding core logic code. This business scenario category is used to indicate which business goal the corresponding core logic code mainly serves when the system is running, such as database reading and writing, file processing, or network request forwarding. After determining the business scenario category, the core logic code belonging to the same business scenario category is grouped into the same business scenario group.

[0047] For the core logic code in each business scenario group, identify the call location recorded in the third-party library call log.

[0048] The call location refers to the specific location in the core logic code where a third-party library call actually occurs, such as the source file path, function name, and the line number where the call statement is located. By matching the core logic code with the call information in the third-party library call record, it is determined which third-party library the corresponding core logic code actually called, where the call occurred, and which code statement the call occurred in.

[0049] Extract the function name and parameter passing path from the call location and combine them into a secure call feature.

[0050] The function name refers to the name of the third-party library function or interface method actually called at the call location, such as the name of the function that performs a database query, the name of the file read / write function, the name of the deserialization function, the name of the script execution function, or the name of the encryption / decryption function. The parameter passing path refers to the path information of where the input parameters of the called function are generated, through which variables are assigned values ​​or intermediate passing nodes, and finally passed to the called function, including, for example, the source variables of the parameters, the assignment statements, the function formal parameters, and the location of the final call arguments.

[0051] By combining the function name and parameter passing path, a secure call feature is obtained that reflects both the call action itself and the source and passing process of the call parameters.

[0052] Methods for performing interference removal on secure call characteristics include: For each secure call feature, obtain a set of security verification patterns that match that secure call feature.

[0053] The security verification mode set refers to a set of protection modes predefined based on historical experience under system security operation for a certain type of security call feature, used to constrain the risk of that call feature. For example, when the security call feature involves database query calls, the corresponding security verification mode may include actions such as parameterization, sensitive character filtering, and input length limitation. When the security call feature involves network request or command execution calls, the corresponding security verification mode may include actions such as permission verification, whitelist verification, and timeout handling.

[0054] The boundary handling code corresponding to the secure call feature is matched with the corresponding set of security verification patterns, and the verification coverage of the boundary handling code to the corresponding set of security verification patterns is calculated.

[0055] The formula for calculating the verification coverage rate is as follows: ;in Indicates the verification coverage; This represents the total number of verification modes in the set of security verification modes corresponding to a certain security call feature; This indicates the number of verification patterns actually covered in the boundary handling code. This value is obtained by checking each verification pattern in the security verification pattern set one by one to determine whether the boundary handling code contains a corresponding statement; for example, the characteristic keyword corresponding to null value verification is... or Then, check each line of code in the boundary handling code, identify whether there are keywords for corresponding null value checks, and count the number.

[0056] Verification coverage is used to indicate the actual degree of protection that boundary handling code provides for the corresponding security call feature. The larger the value, the more likely that the security call feature has been fully constrained and protected by the outer defensive code statements.

[0057] When the verification coverage is higher than or equal to the preset integrity threshold, it is determined that the security call feature is effectively protected, and the corresponding security call feature is removed as an interference item.

[0058] The preset integrity threshold is used to limit the minimum requirements for the boundary processing code to have sufficient protection for the security call feature. For example, if the preset integrity threshold is set to 80%, when the verification coverage is higher than or equal to the preset integrity threshold, it means that the corresponding security call feature has relatively complete input verification or other protection logic on the periphery. If the security call feature is effectively protected, it may be easy to misjudge the protected call as a high-risk call in the subsequent risk analysis. Therefore, such security call features are removed as interference items.

[0059] When the verification coverage is lower than the preset integrity threshold, the secure call characteristics are retained.

[0060] When the verification coverage is lower than the preset integrity threshold, it indicates that the boundary processing code does not fully protect the security call feature. There may be some required security verification modes that are not implemented in the outer code. For example, the call location may have null value verification but lack error return control. In this case, although the security call feature is not fully exposed, there is still a potential risk. Therefore, security call features with verification coverage lower than the corresponding threshold are retained to avoid omissions due to overly lenient judgment.

[0061] By integrating the security call features retained from all business scenario groups, a risk feature set for the corresponding business scenario group is obtained.

[0062] After removing interference items, several security call features will remain in each business scenario group. The retained security call features of each business scenario group will be combined to obtain a risk feature set corresponding to the business scenario.

[0063] For each business scenario group, the risk feature set is normalized to output a security risk indicator sequence. Time-limit fitting is performed on the security risk indicator sequence to locate risk inflection points. Methods include: Obtain the preset risk level corresponding to each risk feature in the risk feature set.

[0064] The preset risk level refers to the risk severity label predefined for different types of risk characteristics, which is used to indicate the degree of impact of the risk characteristic on the source code security. Different risk characteristics can be divided into different levels based on currently known security risk types or security governance experience, such as level 1 risk, level 2 risk and level 3 risk.

[0065] Simultaneously, based on the code commit records, the commit timestamps of each core logic code in the business scenario group corresponding to the risk feature set are identified, and the risk features in the same business scenario group are time-aligned according to their respective commit timestamps.

[0066] The submission timestamp refers to the specific time point when each core logic code enters the version iteration process in the code submission record. By identifying the submission timestamps of all risk features belonging to the same business scenario group, and unifying these submission timestamps into a pre-set complete timeline in this embodiment, each risk feature can be uniquely located on the timeline, thereby achieving time alignment, providing a constraint basis for subsequent operations, and avoiding time sequence confusion of risk features from different time sources in the same business scenario group.

[0067] At each submission timestamp, the sum of risk levels of all risk features in each business scenario group is calculated, and the ratio of this sum of risk levels to the number of risk features in the business scenario group is calculated to obtain the normalized risk value of the corresponding business scenario group.

[0068] In this embodiment, under the same submission timestamp, there may be multiple risk features in the same business scenario group. Therefore, the preset risk level of all risk features in each business scenario group is identified. These preset risk levels are first mapped to a unified risk level standard and then summed to obtain the total risk level. The ratio of the total risk level to the number of risk features in the corresponding business scenario group is calculated, and the normalized risk value of the business scenario group is output to represent the overall risk level of the business scenario group, so that the risk levels of different business scenario groups on the same time axis are comparable.

[0069] The normalized risk values ​​under each submission timestamp are arranged according to the order of submission timestamps to obtain a sequence of security risk indicators.

[0070] The normalized risk values ​​corresponding to each submission timestamp are arranged in chronological order to obtain a security risk indicator sequence. This sequence is used to represent the dynamic trend of risk level changes of the project under test in continuous version iterations.

[0071] A sliding time window is constructed to divide the sequence of safety risk indicators, and the risk mean of the normalized risk value in each sliding time window is calculated.

[0072] In this embodiment, since the frequency of code commit records during version iteration is uneven, there may be periods of dense commits and periods of sparse commits. Therefore, a sliding time window is constructed to divide the security risk indicator sequence, ensuring that each sliding time window covers a continuous time range. The specific window size is set based on the length of the complete time axis, and it is necessary to ensure that the sliding time window can be used to divide the complete time axis an integer number of times. There are several commit timestamps within the time period covered by each sliding time window. Therefore, the mean risk value is obtained by calculating the mean of the normalized risk values ​​under all commit timestamps within the time period.

[0073] Calculate the difference between the average risk values ​​of adjacent sliding time windows. If the difference is greater than a preset turning point threshold, then the boundary time point between adjacent sliding time windows is taken as the risk turning point.

[0074] In this embodiment, a preset turning point threshold is set based on the statistical difference between the average risk values ​​of adjacent sliding time windows in the historical records. This value is used to filter continuous time periods in which the risk level changes significantly. If the difference between the average risk values ​​of adjacent sliding time windows is greater than the preset turning point threshold, the boundary time point between adjacent sliding time windows is taken as the risk turning point.

[0075] It should be noted that this embodiment only discusses the abnormal situation of a sudden increase in risk. Therefore, the risk inflection point is used to represent the key time node when the risk suddenly jumps from a stable state to a high level.

[0076] Methods for identifying pseudo-incremental turning points include: Extract the code review logs of the sliding time window where the risk inflection point is located, and identify whether there are review semantics related to code refactoring or code migration in the code review logs.

[0077] This involves extracting code review logs from the time periods covered by adjacent sliding time windows before and after the risk inflection point, and identifying whether there are review semantics related to code refactoring or code migration in these code review logs. Here, review semantics refers to semantic descriptions in the code review logs that can represent structural reorganization behavior, the most representative of which are code refactoring and code migration.

[0078] Obtain the first set of risk features within the preceding sliding time window and the second set of risk features within the following sliding time window for each risk inflection point.

[0079] In this embodiment, all risk features within the preceding sliding time window adjacent to each risk inflection point are taken as the first risk feature set, and all risk features within the following sliding time window are taken as the second risk feature set, which serve as the data basis for subsequent operations.

[0080] Identify the intersection risk features between the first risk feature set and the second risk feature set, and calculate the risk feature overlap rate by the ratio of the number of intersection risk features to the total number of risk features in the second risk feature set.

[0081] Intersecting risk features refer to risk features that appear simultaneously in both the first and second risk feature sets, i.e., risk features that exist in both time periods before and after the risk inflection point. The risk feature overlap rate is obtained by calculating the ratio of the number of intersecting risk features to the total number of risk features in the second risk feature set. If this value is large, it indicates that most of the risk features in the later sliding time window are not newly generated, but rather a continuation of risks that already existed in the previous sliding time window; conversely, the smaller the value, the more new risk features exist in the later sliding time window.

[0082] When there are review semantics related to code refactoring or code migration, and the overlap rate of risk features is higher than the preset overlap threshold, the corresponding risk inflection point is determined to be a pseudo-sudden increase inflection point.

[0083] If there are review semantics related to code refactoring or code migration, it indicates that code structure adjustments occurred during the time period of the corresponding risk inflection point; at the same time, if the risk feature overlap rate is higher than the preset overlap threshold set based on historical pseudo-sudden increase inflection point judgment experience, it indicates that most of the risk features in the later sliding time window already existed in the previous sliding time window.

[0084] If both conditions are met, it means that although the corresponding risk inflection point appears to be a sudden increase in risk in terms of numerical value, it is more likely due to the redistribution of risk caused by code refactoring or code migration, rather than the large-scale introduction of new risks. Therefore, this risk inflection point is judged as a pseudo-sudden increase inflection point.

[0085] Methods for constructing real risk change paths include: Replace the normalized risk value at the spurious burst inflection point with the mean of the normalized risk values ​​of the previous sliding time window.

[0086] Since pseudo-increment inflection points are essentially numerical jumps caused by risk redistribution due to code refactoring or reordering, rather than an increase in the actual risk level, the normalized risk value of the pseudo-increment inflection point is replaced with the mean of the normalized risk value of the previous sliding time window, so that the risk value of that point is brought back to the average level before refactoring, thus eliminating the pseudo-increment inflection point and removing the impact of the pseudo-increment.

[0087] At the same time, other risk inflection points that were not identified as pseudo-surge inflection points are retained as real risk inflection points.

[0088] Besides the time points that were identified as pseudo-surge inflection points, there are still multiple risk inflection points. Therefore, after eliminating pseudo-surge inflection points, it is necessary to retain the remaining ordinary risk inflection points that were not identified as pseudo-surge inflection points and treat these risk inflection points as real risk inflection points.

[0089] By using adjacent real risk inflection points as dividing boundaries, the complete timeline is divided into multiple continuous risk stages, which are then spliced ​​together in chronological order to obtain the real risk change path.

[0090] The real risk inflection points are the selected effective decomposition points. Adjacent real risk inflection points correspond to a continuous time interval with relatively consistent risk change trends. Therefore, adjacent real risk inflection points are used as the dividing boundary to divide the complete time axis into several continuous risk stages. Each continuous risk stage corresponds to a risk change process. The real risk change path is obtained by splicing them together in chronological order.

[0091] Methods for extracting stage-specific risk features based on actual risk change paths and then reducing and compressing these features to output low-dimensional risk feature vectors include: The mean risk of a stage is obtained by calculating the mean of the normalized risk values ​​of all sampling points in each continuous risk stage in the actual risk change path.

[0092] Each continuous risk stage contains multiple sampling points corresponding to submission timestamps, and each sampling point has a normalized risk value. The average risk of each stage is obtained by calculating the mean of the normalized risk values ​​of all sampling points in each continuous risk stage, which reflects the overall risk level of the corresponding continuous risk stage.

[0093] The maximum number of affected codes in each continuous risk phase is identified as the peak number. The average risk of the corresponding continuous risk phase and the peak number are combined to form the basic characteristics of that continuous risk phase.

[0094] The maximum number of affected code refers to the maximum number of lines of code that have been changed in the code commit records corresponding to all commit timestamps in a certain continuous risk phase. It is used to represent the maximum scale of the impact of a single code commit on the source code in that continuous risk phase, and this number is used as the peak number. The average risk of the phase and the peak number are combined to form the basic characteristics of the corresponding continuous risk phase that can simultaneously reflect the risk level of the phase and the scope of code changes in the phase.

[0095] The rate of change of the average risk of adjacent consecutive risk stages and the difference in the peak value of the stage are calculated and combined to form the stage correlation feature.

[0096] The method involves calculating the rate of change of the average risk level and the difference in the peak value between adjacent consecutive risk stages, which are combined to form a stage correlation feature. This feature is used to represent the dynamic change trend between two adjacent consecutive risk stages, that is, the direction and magnitude of change in risk level and scope of influence along the stage.

[0097] The basic features and stage-related features of each continuous risk stage are concatenated to obtain the stage risk features. Specifically, the basic features and stage-related features corresponding to each continuous risk stage are concatenated and merged into a unified feature vector, which is the stage risk feature.

[0098] Remove feature items in the stage risk features whose rate of change of the stage risk mean is lower than a preset rate of change threshold, and transform the adjusted stage risk features into a low-dimensional risk feature vector.

[0099] The system sets a preset threshold for the rate of change of the average risk value of a large number of adjacent consecutive risk stages in historical records. This threshold is used to identify situations where the risk level changes significantly. Since retaining all feature items of all consecutive risk stages after constructing the stage risk features can easily result in high-dimensional features, which is not conducive to the subsequent processing by the preset lightweight risk prediction model, a threshold comparison is used for screening. If the rate of change of the average risk value of a certain consecutive risk stage is low, it means that the risk level of this stage has hardly changed compared to the previous stage, and it belongs to a stable transition stage with little impact on the overall risk level. Therefore, such feature items are removed, thereby reducing the feature dimensionality and improving the processing efficiency of the subsequent model.

[0100] In this embodiment, the preset lightweight risk prediction model refers to a low-parameter risk prediction model pre-trained based on historical project samples. Here, a logistic regression model is used as the main body of the model, and the low-dimensional risk feature vector is used as the input data of the model. The logistic regression model is used to infer the input data and output a security risk score. Since the low-dimensional risk feature vector has been reduced in dimensionality, the logistic regression model can complete risk inference and prediction with low computational overhead, which meets the requirements of high-frequency execution and fast feedback of source code security detection in version iteration scenarios.

[0101] Methods for generating a list of priority repair recommendations include: Obtain the preset risk level of each risk feature in the corresponding risk feature set and the average risk of the continuous risk stage to which it belongs; By using preset priority weights, the safety risk score, preset risk level, and the average risk of the corresponding continuous risk stage are weighted and summed to obtain the repair priority score for each risk feature.

[0102] In this embodiment, the safety risk score, the preset risk level, and the average risk of the corresponding continuous risk stage are first normalized to eliminate the difference in dimensions; then, the preset priority weights are used to perform a weighted summation to obtain the repair priority score of the corresponding risk feature.

[0103] It should be noted that the preset priority weights are used to represent the influence of the security risk score, the preset risk level, and the average risk of the corresponding continuous risk stage on the determination of the repair priority. By statistically analyzing the actual repair sequence corresponding to different risk characteristics in historical projects, and analyzing the information contribution of the security risk score, the preset risk level, and the average risk of the stage on the actual repair sequence, values ​​are assigned according to the proportion of the information contribution of the three dimensions, while ensuring that the sum of the preset priority weights of the three dimensions is 1.

[0104] The final fix priority score is used to comprehensively measure the urgency of each risk feature in the current version state, indicating how urgently it needs to be fixed.

[0105] All risk characteristics are sorted based on their repair priority scores to obtain a list of priority repair recommendations.

[0106] The priority repair suggestion list is used to output the sequence of risk characteristics that need to be given priority and dealt with in the current project to be detected. It is used to help those skilled in the art or the system to prioritize the limited repair resources to the risks that need to be dealt with most.

[0107] This embodiment achieves a source code security detection method based on semantic features and a lightweight model by collecting multi-source data required for source code security detection. Compared with existing experience, semantic decomposition eliminates the detection object positioning deviation caused by the mixing of core logic code and boundary processing code; interference elimination distinguishes between calls that appear to have risks but have been effectively protected and calls that actually expose risks, avoiding misjudgment problems in third-party library call scenarios; identifying pseudo-surge inflection points filters out false risk leaps; and dimensionality reduction and compression of features enable the preset lightweight risk prediction model to complete inference and output with lower computational overhead, avoiding the inference burden and feedback lag caused by directly inputting high-dimensional features into the model. Overall, it improves the positioning accuracy, risk identification accuracy, and detection efficiency in version iteration scenarios of source code security detection.

[0108] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

[0109] All formulas in this manual are dimensionless and calculated numerically. The formulas are derived from software simulations based on a large amount of collected data to obtain the most recent real-world results. The preset parameters and thresholds in the formulas are set by those skilled in the art according to the actual situation.

[0110] Although embodiments of this application have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of this application, the scope of which is defined by the claims and their equivalents.

Claims

1. A source code security detection method based on semantic features and a lightweight model, characterized in that, include: S1. Obtain the code commit records, third-party library call records, and corresponding requirement change documents and code review logs of the project to be tested during the version iteration process; S2. Extract semantic features of requirements based on requirement change documents, and extract semantic features of code based on code commit records; Identify the mapping relationship between the semantic features of requirements and the semantic features of code, and perform semantic decomposition on the code submission record based on the mapping relationship, outputting a code structure tuple consisting of core logic code and boundary handling code; S3. Identify business scenario categories for the core logic code and classify them according to these categories to obtain business scenario groups; extract security call features from third-party library call records in each business scenario group; combine the corresponding boundary processing code to perform interference removal on the security call features to obtain the risk feature set of the corresponding business scenario group; S4. Normalize the risk feature set for each business scenario group and output a sequence of security risk indicators; Fit the sequence of safety risk indicators over a time period to pinpoint risk inflection points; Identify the distribution of risk characteristics before and after risk inflection points, and identify pseudo-surge inflection points by combining code review logs; Eliminate pseudo-sudden increase turning points and construct the real risk change path based on the remaining risk turning points; S5. Extract stage risk features based on the actual risk change path, reduce the dimensionality of the stage risk features to output a low-dimensional risk feature vector, input the low-dimensional risk feature vector into a preset lightweight risk prediction model to obtain a safety risk score, and generate a priority repair suggestion list by combining the corresponding risk feature set.

2. The source code security detection method based on semantic features and a lightweight model according to claim 1, characterized in that, The methods for identifying the mapping relationship between requirement semantic features and code semantic features include: The requirement change document is segmented into words to extract the set of action words and the set of business nouns. The set of action words and the set of business nouns are then transformed into requirement feature vectors as requirement semantic features. Obtain the abstract syntax tree corresponding to the code commit record, identify the changed strings and instructions, and combine them into a code feature vector as the code semantic features; Calculate the cosine similarity between the requirement feature vector and each code feature vector, and use this cosine similarity as the mapping relationship between the corresponding requirement semantic features and code semantic features.

3. The source code security detection method based on semantic features and a lightweight model according to claim 2, characterized in that, The methods for semantically splitting code commit records include: Set core judgment thresholds and boundary judgment thresholds, and ensure that the core judgment threshold is greater than the boundary judgment threshold; When the cosine similarity of the mapping relationship is higher than the core judgment threshold, the corresponding code submission record is used as the core logic code; if the cosine similarity is lower than the boundary judgment threshold, and the corresponding code submission record is used to handle abnormal situations, the code submission record is used as the boundary handling code. When the cosine similarity corresponding to the mapping relationship is between the core judgment threshold and the boundary judgment threshold, the corresponding code submission record is divided into code statements to form a code statement sequence; the code feature vector of each code statement in the code statement sequence is constructed, the cosine similarity between each code feature vector and the corresponding requirement feature vector is calculated, and threshold judgment is performed one by one; All core logic code and boundary handling code are combined into code structure tuples.

4. The source code security detection method based on semantic features and a lightweight model according to claim 3, characterized in that, The process involves identifying business scenario categories for the core logic code and classifying them according to these categories to obtain business scenario groups. Within each business scenario group, the methods for extracting security call characteristics from third-party library call records include: Extract the set of operation keywords from the core logic code, identify the number of characters in the intersection of the set of operation keywords and the set of template keywords for each preset scenario, and calculate the ratio of the number of characters in the intersection to the total number of characters in the set of operation keywords to obtain the template matching degree; The scenario category corresponding to the scenario with the highest template matching degree is taken as the business scenario category, and all core logic code belonging to the same business scenario category is aggregated to obtain business scenario grouping; For the core logic code in each business scenario group, identify the call location recorded in the third-party library call record; extract the call function name and parameter passing path at the call location, and combine them into a secure call feature.

5. The source code security detection method based on semantic features and a lightweight model according to claim 4, characterized in that, The methods for performing interference removal on secure call features include: For each security call feature, obtain the set of security verification patterns that match the security call feature; perform pattern matching between the boundary processing code corresponding to the security call feature and the corresponding set of security verification patterns, and calculate the verification coverage of the boundary processing code to the corresponding set of security verification patterns; When the verification coverage is higher than or equal to the preset integrity threshold, it is determined that the security call feature is effectively protected, and the corresponding security call feature is removed as an interference item. When the verification coverage is lower than the preset integrity threshold, the security call features are retained; the security call features retained in all business scenario groups are integrated to obtain the risk feature set of the corresponding business scenario group.

6. The source code security detection method based on semantic features and a lightweight model according to claim 5, characterized in that, The risk feature set for each business scenario group is normalized to output a sequence of security risk indicators. Methods for fitting time periods to a sequence of safety risk indicators to pinpoint risk inflection points include: Obtain the preset risk level corresponding to each risk feature in the risk feature set; at the same time, identify the submission timestamp of each core logic code in the business scenario group corresponding to the risk feature set based on the code submission record, and align the risk features in the same business scenario group according to their respective submission timestamps; At each submission timestamp, calculate the sum of risk levels of all risk features in each business scenario group, and calculate the ratio of the sum of risk levels to the number of risk features in the business scenario group to obtain the normalized risk value of the corresponding business scenario group. The normalized risk values ​​under each submission timestamp are arranged according to the order of submission timestamps to obtain a sequence of security risk indicators. A sliding time window is constructed to divide the sequence of safety risk indicators, and the risk mean of the normalized risk value in each sliding time window is calculated; Calculate the difference between the average risk values ​​of adjacent sliding time windows. If the difference is greater than a preset turning point threshold, then the boundary time point between adjacent sliding time windows is taken as the risk turning point.

7. The source code security detection method based on semantic features and a lightweight model according to claim 6, characterized in that, The methods for identifying pseudo-incremental inflection points include: Extract the code review logs of the sliding time window where the risk inflection point is located, and identify whether there are review semantics related to code refactoring or code migration in the code review logs; Obtain the first risk feature set within the preceding sliding time window and the second risk feature set within the following sliding time window for each risk inflection point; identify the intersection risk features between the first and second risk feature sets, and calculate the risk feature overlap rate by the ratio of the number of intersection risk features to the total number of risk features in the second risk feature set. When there are review semantics related to code refactoring or code migration, and the overlap rate of risk features is higher than the preset overlap threshold, the corresponding risk inflection point is determined to be a pseudo-sudden increase inflection point.

8. The source code security detection method based on semantic features and a lightweight model according to claim 7, characterized in that, The methods for constructing real risk change paths include: Replace the normalized risk value at the pseudo-increase inflection point with the mean of the normalized risk value of the previous sliding time window; at the same time, retain other risk inflection points that were not identified as pseudo-increase inflection points as true risk inflection points. By using adjacent real risk inflection points as dividing boundaries, the complete timeline is divided into multiple continuous risk stages, which are then spliced ​​together in chronological order to obtain the real risk change path.

9. The source code security detection method based on semantic features and a lightweight model according to claim 8, characterized in that, The method of extracting stage risk features based on the actual risk change path and then reducing and compressing these stage risk features to output a low-dimensional risk feature vector includes: The mean risk of a stage is obtained by calculating the mean of the normalized risk values ​​of all sampling points in each continuous risk stage in the actual risk change path. The maximum number of affected codes in each continuous risk phase is identified as the peak number, and the average risk of the corresponding continuous risk phase and the peak number are combined to form the basic characteristics of that continuous risk phase. Calculate the rate of change of the average risk of adjacent consecutive risk stages and the difference of the peak values ​​of the stages, and combine them into stage correlation features; The basic features and stage-related features of each continuous risk stage are concatenated to obtain the stage risk features; feature items whose rate of change of the stage risk mean is lower than the preset rate of change threshold are deleted from the stage risk features, and the adjusted stage risk features are transformed into a low-dimensional risk feature vector.

10. The source code security detection method based on semantic features and a lightweight model according to claim 9, characterized in that, The methods for generating the priority repair suggestion list include: Obtain the preset risk level and the average stage risk of the continuous risk stage to which each risk feature belongs in the corresponding risk feature set; use preset priority weights to perform a weighted summation of the safety risk score, preset risk level and the average stage risk of the continuous risk stage to which it belongs, to obtain the repair priority score of each risk feature; sort all risk features based on the size of the repair priority score to obtain a priority repair suggestion list.

Images