Source code security detection large model dimensionality reduction method based on homomorphic merging

Abstract

The invention relates to the technical field of large model dimension reduction. The invention relates to a source code security detection large model dimensionality reduction method based on homomorphic merging. The method comprises the following steps: S1, collecting source code data from a development environment, and obtaining performance data of a computer at the same time; s2, carrying out data feature extraction analysis on the source code data, carrying out token series conversion on the source code data by data features according to an analysis result, and carrying out position code distribution on each token at the same time; according to the method, similar or repeated tokens in source codes are merged into a unified representation through a homomorphic merging technology, redundant information in the source codes is effectively reduced in the process, the dimension of data is remarkably reduced, the process is crucial for a large-scale source code security detection model, and due to the fact that a code library usually contains thousands of tokens, the security of the source codes is greatly improved. And the combined data set is more compact.

Description

Technical Field

[0001] The present invention relates to the technical field of large model dimensionality reduction, and in particular to a large model dimensionality reduction method for source code security detection based on homomorphic merging. Background Art

[0002] Currently, artificial intelligence models are developing in the direction of large-scale and ultra-large-scale models, such as GPT-4, which has hundreds of billions or even trillions of parameters. More tokens mean more data that needs to be processed and learned by the model, and the training of large-scale models itself requires strong computing power support to complete complex matrix operations and back-propagation operations. The development of multimodal artificial intelligence requires models to process multiple types of data such as text, code, functions, processes, audio, source code, etc. at the same time. The number of tokens in source code and high-resolution code, functions, processes and other data usually far exceeds that of text. For example, a few minutes of source code may contain millions of tokens. When processing large amounts of multimodal data, the model needs to consume more computing power to encode, decode and fuse data of different modalities to extract valuable information, which increases computing power requirements dramatically; In order to meet the growing demand for computing power, enterprises and research institutions need to purchase a large number of high-performance computing chips. In order to maintain competitiveness, they need to continuously update and upgrade chips, which makes the hardware acquisition cost continue to increase. In addition to chips, a large number of servers and storage devices are also needed to support the operation of computing power and data storage. With the increase in data volume and computing tasks, the scale of servers and storage devices also needs to be continuously expanded, which further increases the hardware cost investment. At the same time, as the scale of the project increases, the amount of code grows exponentially. The static analysis and dynamic analysis methods currently used may face performance bottlenecks, especially static analysis methods. As the amount and complexity of code increase, it may lead to a large number of false positives or false negatives, which in turn affects the detection efficiency and accuracy. Therefore, a dimensionality reduction method for large source code security detection models based on homomorphic merging is proposed. Summary of the invention

[0003] The purpose of the present invention is to provide a source code security detection large model dimensionality reduction method based on homomorphic merging to solve the problems raised in the above background technology.

[0004] To achieve the above purpose, a source code security detection large model dimensionality reduction method based on homomorphic merging is provided, which includes the following steps: S1. Collect source code data from the development environment and obtain computer performance data; S2. Extract and analyze the data features of the source code data, and convert the source code data into a token series based on the data features according to the analysis results, and assign a position code to each token; S3. Perform redundancy detection on all tokens to extract the same tokens for homomorphic merging. Replace the multiple tokens after homomorphic merging with a new token for representation, and then establish a token sequence library to collect the tokens. S4. Calculate the similarity values ​​of the data features of all tokens in the token sequence library, perform adaptive analysis of the number of tokens based on the complexity of the performance data and source code data, and set a dynamic similarity threshold based on the analysis results; S5. The similarity values ​​between tokens are combined with the dynamic similarity threshold for screening, and then the tokens that meet the load dynamic similarity threshold are homomorphically merged according to the screening results. At the same time, the data dimension reduction analysis is performed on the tokens after homomorphic merger to generate representative feature data; S6: Feed the token processed by S5 back to the token sequence library for compression and replacement, then input the token sequence library into the source code model for training, and adjust the dynamic similarity threshold according to the training results.

[0005] As a further improvement of the technical solution, S1 collects source code data from different development environments and formats and adjusts the source code data, thereby unifying the format of the source code data.

[0006] As a further improvement of the technical solution, the S1 is connected to the management port of the computer, thereby extracting the performance data of the computer in real time in the management port, thereby monitoring the load status of the computer.

[0007] As a further improvement of the technical solution, the steps of S2 are as follows: S2.1. Extract features from the source code data to obtain static features and semantic features of the source code data, then aggregate the static features and semantic features into data features, and perform security detection on the source code data based on the data features, and only retain the source code data whose data features pass the security detection; S2.2, convert the source code data retained in S2.1 into a token series with data features, where the token series consists of multiple tokens, each of which represents a basic unit, function name, function block, and process in the source code; S2.3. Assign a position code to each token so that each token corresponds to a separate position code. The position code is used to represent the application of the token, thereby describing the source code data as a series of token reference codes.

[0008] As a further improvement of the technical solution, the steps of S3 are as follows: S3.1. Use the redundancy detection algorithm to perform redundancy detection on all tokens to extract the same token, and then homomorphically merge multiple consecutive combinations of repeated tokens to form a new token. S3.2. Establish a token sequence library, and use the token sequence library to collect the tokens reorganized in S3.1 and the tokens that have not been merged.

[0009] As a further improvement of the technical solution, the step of S4 is as follows: S4.1. Extract the data features of all tokens in the token sequence library, and then calculate the similarity values ​​of the data features of all tokens to obtain the similarity values ​​between each token; S4.2. Combine the source code data corresponding to all tokens in the token sequence library to perform complexity numerical analysis, obtain the complexity value of the token sequence library based on the analysis results, and then combine the performance data with the complexity value to perform token quantity adaptation analysis, obtain the number of tokens that the performance data can adapt to based on the analysis results, and set the number of tokens obtained from the analysis as the dynamic similarity threshold.

[0010] As a further improvement of the present technical solution, when the latest performance data in S4.2 shows that the load status of the computer is overloaded, the latest performance data is substituted into S4.2 to reset the dynamic similarity threshold; conversely, when the latest performance data shows that the load status of the computer is running normally, monitoring is continued.

[0011] As a further improvement of the technical solution, the step of S5 is as follows: S5.1. Filter the similarity values ​​between tokens in combination with the dynamic similarity threshold, so as to form a token merge list that meets the dynamic similarity threshold according to the similarity value composition, and then calculate the average similarity value of the obtained token merge list, and give priority to retaining the token merge list with the highest average similarity value; S5.2. All tokens are homomorphically merged according to the token merge list retained in S5.1. At the same time, data features are extracted from the homomorphically merged tokens. The extracted data features are then combined for feature selection and data dimension reduction analysis. According to the analysis results, corresponding representative feature data is generated for the homomorphically merged tokens.

[0012] As a further improvement of the technical solution, the step of S6 is as follows: S6.1, feed back the tokens after homomorphic merging in S5.2 to the token sequence library for compression and replacement, so that the token sequence library uses the latest compressed and replaced tokens as the training set; S6.2. Input the training set into the source code model for training, and adjust the dynamic similarity threshold according to the training results. Then, the training results are fed back to the developer. When the developer's feedback result is to continue adjustment, the dynamic similarity threshold is increased.

[0013] Compared with the prior art, the present invention has the following beneficial effects: 1. In this large-scale source code security detection model dimensionality reduction method based on homomorphic merging, similar or repeated tokens in the source code are merged into a unified representation through homomorphic merging technology. This process effectively reduces redundant information in the source code and significantly reduces the dimension of the data. This is crucial for large-scale source code security detection models, because the code base usually contains tens of thousands of tokens. The merged data set is more compact, which reduces the computational complexity of model training and reasoning, improves the efficiency of data processing, enables the source code model to complete training in a shorter time, and can process large-scale code bases more efficiently.

[0014] 2. In this dimensionality reduction method for source code security detection large models based on homomorphic merging, the dynamic similarity threshold adjustment mechanism allows the token merging strategy to be dynamically optimized according to training results and developer feedback. If the developer feedback requires stricter or looser similarity requirements for certain tokens, the threshold adjustment can help the system handle boundary fuzzy situations more accurately when merging tokens. This dynamic adjustment ensures that the model is more flexible in dealing with different types of source code, and can adjust its merging and identification strategies according to specific code features. Through this dynamic similarity threshold adjustment, the source code security detection large model can continuously optimize itself, adapt to different code styles and security requirements, and improve the flexibility and adaptability of the detection process.

[0015] 3. This large model dimensionality reduction method for source code security detection based on homomorphic merging effectively improves the efficiency, accuracy and scalability of source code security detection by combining technologies such as homomorphic merging, feature extraction, feature selection, dimensionality reduction and dynamic similarity threshold adjustment. By reducing redundancy, improving model generalization ability, flexibly adjusting similarity thresholds, reducing model complexity, improving detection accuracy and expanding coverage, this system can help developers discover potential security vulnerabilities in a shorter time, reduce the time and cost of vulnerability repair, and ultimately enhance the security of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is the overall flow chart of the present invention; Figure 2 A flowchart of the source code data that retains data features and passes security inspection for the present invention; Figure 3 A flowchart of the present invention for making multiple repeated tokens form a new token; Figure 4 A flowchart of obtaining similar values ​​between each token in the present invention; Figure 5 A flowchart of the present invention for generating corresponding representative feature data for homomorphically merged tokens according to the analysis results; Figure 6 The present invention is a flowchart of using the latest compressed and replaced tokens in the token sequence library as a training set. DETAILED DESCRIPTION

[0017] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0018] See also Figure 1 As shown, the purpose of this embodiment is to provide a source code security detection large model dimensionality reduction method based on homomorphic merging, including the following steps: S1. Collect source code data from the development environment and obtain computer performance data; S1 collects source code data from different development environments and formats and adjusts the source code data to make the format of the source code data uniform. The specific steps are as follows: Source code data collection: Identify the development environment or version control system where the source code comes from, obtain source code files from different environments, and collect source code related metadata in addition to the code files themselves; Remove useless comments and blank characters: Use regular expressions or specialized code parsing tools to remove comments in the code, remove extra spaces, tabs, blank lines, and line breaks to ensure the standardization of the code; Unified code and naming standards: Ensure that all variables, functions, and class names comply with unified naming rules, ensure that the definition position of functions and classes in the code, parameter order, etc. comply with unified standards, and unify the comment format. Each function should have a clear documentation string, and then use tools to automatically format the source code to ensure a unified code style.

[0019] S1 accesses the management port of the computer, and extracts the performance data of the computer in real time from the management port, so as to monitor the load status of the computer. The steps are as follows: Get performance data through the management port: Use the management port to get computer performance data directly from the hardware level, including CPU usage, memory usage, hard disk health, power management status, etc. Load status monitoring and real-time extraction: The process of extracting performance data in real time is a continuous process, so it is necessary to set up automated scripts and collect data regularly to monitor the computer's load status. Using Python scripts or other programming languages, combined with scheduled tasks (such as cron), you can regularly capture data and save it to log files or databases. For example, you can extract performance data every 5 minutes and calculate the load index.

[0020] S2. Extract and analyze the data features of the source code data, and convert the source code data into a token series based on the data features according to the analysis results, and assign a position code to each token; like Figure 2 As shown, the steps of S2 are as follows: S2.1. Extract features from the source code data to obtain static features and semantic features of the source code data. Then aggregate the static features and semantic features into data features. Perform security detection on the source code data based on the data features. Only source code data with data features that pass the security detection is retained. The steps are as follows: Static feature extraction: For lexical features, the basic structure information of the code is obtained through keywords, identifiers, constants, operators, etc. in the code. For grammatical structure features, the function, variable definition, conditional statement, loop structure, etc. in the code are used. For control flow features, the control flow information of the program is obtained by analyzing the control flow graph, such as determining whether there are potential risks such as dead loops and callbacks, and then extracting structural features such as function calls, variable usage, and expressions through the analysis of abstract syntax trees. For dependency relationships, the calling relationship, variable dependency, data flow, etc. between modules or functions in the code are analyzed; Semantic feature extraction: For data flow analysis, extract the data flow between variables in the code, analyze the process of variable transfer, modification, and use in the function, and identify potential vulnerabilities or risks. For control flow analysis, analyze the execution path of the program according to the control flow graph to identify whether there are potential logical errors. For sensitive operation detection, extract specific system calls, network access, database operations, file reading and writing, and other sensitive operations to check whether there are possible security risks in the source code. Feature aggregation: Integrate static features and semantic features to form a unified feature set; Security testing: Security testing can be performed through different machine learning algorithms or rule libraries, using labeled code data sets (including secure and non-secure source code), training models based on feature sets, training models and evaluating their performance. Cross-validation and other methods can be used to evaluate the accuracy, precision, recall rate, etc. of the model, and then the trained model is used to perform security analysis on the source code to be tested. The security of the source code is judged based on the prediction results. Finally, based on the security test results, the source code that has passed the security test is retained and the source code with security risks is eliminated.

[0021] S2.2, convert the source code data retained in S2.1 into a token series with data features, where the token series consists of multiple tokens, each of which represents a basic unit, function name, function block, and process in the source code; Decompose the source code into basic grammatical units. In the code, each token represents a basic unit in the source code, such as a keyword, identifier, operator, constant, etc., thereby converting the source code into a token sequence; S2.3. Assign position codes to each token so that each token corresponds to a separate position code. The position code is used to represent the application of the token, thereby describing the source code data as a series of token reference codes. The steps are as follows: Positional encoding: In order for the model to understand the order and structure of tokens, each token needs to be assigned a positional encoding. Positional encoding is achieved by mapping the position of the token to a vector space. Common positional encoding methods are based on the calculation of sine and cosine functions. The most common one is the encoding method used in the Transformer model. Positional encoding and token fusion: After position encoding, each token will be fused with its original representation (such as word embedding). Word embedding is the process of mapping tokens to vector space, usually using pre-trained word vector models or code-specific embedding methods; Create a Token Series: The whole process finally generates a sequence represented by multiple tokens, and each token has a positional code. These token sequences will be used as input for the deep learning model.

[0022] S3. Perform redundancy detection on all tokens to extract the same tokens for homomorphic merging. Replace the multiple tokens after homomorphic merging with a new token for representation, and then establish a token sequence library to collect the tokens. like Figure 3 As shown, the steps of S3 are as follows: S3.1. Use the redundancy detection algorithm to perform redundancy detection on all tokens to extract the same token, and then homomorphically merge multiple consecutive repeated tokens to form a new token. The steps are as follows: Token frequency statistics: Count the frequency of each token. Through the frequency table, you can understand which tokens are repeated and prepare for redundancy detection. The formula is as follows: ; Among them, f(t) represents the frequency of the token, that is, the number of times the token appears in all texts, t is a single token, I(t, x i ) is an indicator function, indicating that in the text x i Whether t appears in the text, if it appears, the value is 1, if it does not appear, the value is 0, and n is the total number of texts; Identify redundant tokens: Based on the frequency table, identify those tokens that appear frequently. These tokens are redundant tokens. During the redundancy detection process, tokens whose frequency exceeds a certain threshold will be marked as redundant. For example, the threshold can be set to 2. If a token appears more than the threshold (twice), it is considered redundant. Combine redundant tokens: For repeated tokens, merge them. The merge operation reduces redundant representation by converting repeated tokens into a new token. The formula is as follows: ; Among them, t new is the new merged token, merge is the merge function, which combines multiple duplicate tokens into a new token, t1, t 2， t n For repeated tokens; Updating a token means: updating the original data, replacing redundant tokens with merged new tokens, removing redundant data, and making the data more compact.

[0023] S3.2. Establish a token sequence library, and use the token sequence library to collect the tokens reorganized in S3.1 and the tokens that have not been merged. The steps are as follows: Establish a Token Sequence Library: Create an empty token sequence library. Each token can be represented in the library in its string form. Through the tokenization process, the input text data is converted into a token sequence and stored in the token sequence library. At the same time, each token is stored in the library as a key, and frequency information or other metadata may be attached. If the token already exists, its frequency or other information is updated; The reorganized token is included: Include the tokens reassembled from S3.1, that is, the tokens after redundancy detection and merging, and add them to the token sequence library, and ensure that they can be effectively managed and queried with other unmerged tokens; Collect unmerged tokens: All unmerged tokens (i.e. original tokens) and merged tokens are stored together in the token sequence library to ensure data consistency and integrity, so that the token sequence library contains all merged tokens and original tokens at the same time, and their frequency information is updated and consistent.

[0024] S4. Calculate the similarity values ​​of the data features of all tokens in the token sequence library, perform adaptive analysis of the number of tokens based on the complexity of the performance data and source code data, and set a dynamic similarity threshold based on the analysis results; like Figure 4 As shown, the steps of S4 are as follows: S4.1. Extract the data features of all tokens in the token sequence library, and then calculate the similarity values ​​of the data features of all tokens to obtain the similarity values ​​between each token. The steps are as follows: Data feature extraction: extract the feature vector of each token from the token sequence library; Calculate the similarity between tokens: Calculate the similarity between all tokens in the vector space. The formula is as follows: ; in, are the feature vectors of two different tokens. is the dot product of the vectors, and is the length of the two token feature vectors, is the similarity value of two different tokens. The range of similarity value is [-1,1], where 1 means that the two vectors are exactly the same, -1 means they are completely opposite, and 0 means there is no similarity at all.

[0025] S4.2. Combine the source code data corresponding to all tokens in the token sequence library for complexity numerical analysis. Obtain the complexity value of the token sequence library based on the analysis results. Then combine the performance data with the complexity value to perform token quantity adaptation analysis. Obtain the number of tokens that the performance data can adapt to based on the analysis results. At the same time, set the number of tokens obtained from the analysis as the dynamic similarity threshold. The steps are as follows: Token sequence library complexity numerical analysis: It is necessary to extract the source code data of each token from the token sequence library. The source code data can be text data or related program execution or resource usage data. The main purpose of complexity numerical analysis is to measure the complexity of each token sequence. This can be done by calculating some common complexity indicators, such as entropy, redundancy, information content, etc., to quantify the complexity of the sequence; Performance data combined with complexity values ​​for token quantity adaptability analysis: When performing performance analysis, it is necessary to evaluate the relationship between the number of tokens and system performance. Specifically, the number of tokens has an impact on system performance (such as running speed, resource usage, etc.). When analyzing complexity, performance indicators such as memory usage and execution time will change with the change of token sequence. Therefore, it is necessary to combine complexity values ​​and performance data for adaptability analysis; Get the number of tokens that can be adapted to the performance data: Determine the maximum acceptable performance degradation range based on the performance data and complexity analysis, and get the corresponding number of tokens based on the adaptability analysis results; Setting dynamic similarity threshold: Based on the above analysis, the final number of tokens needs to be set as the dynamic similarity threshold, so that the optimal number of tokens obtained through adaptive analysis is used as the preliminary threshold; The key to this analysis method is to determine an adaptability parameter through the mathematical relationship between complexity and performance, so that the system can maintain good performance under different numbers of tokens.

[0026] In S4.2, when the latest performance data shows that the load status of the computer is overloaded, the latest performance data is substituted into S4.2 to reset the dynamic similarity threshold. Conversely, when the latest performance data shows that the load status of the computer is running normally, monitoring is continued.

[0027] By monitoring the latest performance data, the current computer load status (overload or normal) is first determined. If the system is found to be overloaded, the dynamic similarity threshold is reset based on the current performance data so that the system can adapt to the changing load conditions. If the load status is normal, the system continues to monitor and does not change the current threshold. In this way, through dynamic adjustment and continuous monitoring, the system can ensure efficient and stable operation.

[0028] S5. The similarity values ​​between tokens are combined with the dynamic similarity threshold for screening, and then the tokens that meet the load dynamic similarity threshold are homomorphically merged according to the screening results. At the same time, the data dimension reduction analysis is performed on the tokens after homomorphic merger to generate representative feature data; like Figure 5 As shown, the steps of S5 are as follows: S5.1. Filter the similarity values ​​between tokens in combination with the dynamic similarity threshold, so as to form a token merge list that meets the dynamic similarity threshold according to the similarity value composition, and then calculate the average similarity value of the obtained token merge list, and give priority to retaining the token merge list with the highest average similarity value. The steps are as follows: Token screening based on dynamic similarity threshold: According to the calculated similarity value, the dynamic similarity threshold is set to screen out token pairs that meet the threshold condition and form a new candidate list. This threshold is dynamically adjusted based on the current system or performance status and is used to screen tokens with similarity higher than the threshold. Merge tokens that meet the threshold conditions into a list: The token pairs filtered out above will be merged into a token merge list based on the similarity relationship. The merge here means combining tokens with high similarity to form a new token set. The merged token set should contain all token pairs that meet the conditions; Calculate the average similarity value of each token merge list: For each token merge list, its average similarity will be calculated. The average similarity is the average of the similarities of all token pairs in the list, which represents the overall similarity of the list; Select the token merge list with the highest average similarity value: After obtaining multiple token merge lists and their average similarities, you need to select the merge list with the highest average similarity as the final result. This is to retain the most similar token set to improve the efficiency and accuracy of subsequent operations. The whole process is to calculate the similarity between tokens and filter them with the dynamic similarity threshold, and finally form multiple merged lists of tokens that meet the threshold. In all merged lists, the average similarity of each list is calculated, and the merged lists of tokens with the highest average similarity are retained first. Through this process, the final token set will be the most streamlined and highly similar set, which can improve the efficiency and performance of subsequent calculations.

[0029] S5.2. Homomorphically merge all tokens according to the token merge list retained in S5.1. At the same time, extract data features from the homomorphically merged tokens. Then combine the extracted data features for feature selection and data dimension reduction analysis. Generate corresponding representative feature data for the homomorphically merged tokens according to the analysis results. The steps are as follows: Homomorphic merging of the retained token list: Perform homomorphic merging on the retained token merge list. Homomorphic merging means merging them into a unified representation while maintaining the information of each token. This usually includes unifying similar tokens or merging them according to some rules (such as vectorization). Extract data features from homomorphically merged tokens: Once the homomorphically merged tokens are obtained, data features need to be extracted from these tokens. These features can be various statistical properties of the original tokens, such as frequency, position, part of speech, context information, etc., or vector tables generated by models (such as embedding models); Feature selection and data dimensionality reduction: Since a large number of features may be generated during feature extraction, the next step is to perform feature selection and data dimensionality reduction. The purpose of feature selection is to remove redundant or irrelevant features, while data dimensionality reduction is to reduce the dimension of the feature space through techniques (such as principal component analysis PCA, linear discriminant analysis LDA), thereby making the data easier to process; Generate representative feature data: Based on feature selection and dimensionality reduction, representative feature data can be generated for the homomorphically merged tokens. These representative feature data are the final "feature vector" representation of each token, which is used for subsequent analysis, classification, clustering and other tasks; The tokens are integrated into a unified representation through homomorphic merging, and then relevant features are extracted from these merged tokens. These features will be optimized through feature selection and dimensionality reduction techniques to finally generate representative feature data for each token. This series of operations can reduce redundant data and convert tokens into feature forms that are easier to analyze and process, ensuring the efficiency and accuracy of models and analysis.

[0030] S6: Feed the token processed by S5 back to the token sequence library for compression and replacement, then input the token sequence library into the source code model for training, and adjust the dynamic similarity threshold according to the training results.

[0031] like Figure 6 As shown, the steps of S6 are as follows: S6.1, feed back the tokens after homomorphic merging in S5.2 to the token sequence library for compression and replacement, so that the token sequence library uses the latest compressed and replaced tokens as the training set; S6.2. Input the training set into the source code model for training, and adjust the dynamic similarity threshold according to the training results. Then feed back the training results to the developer. If the developer's feedback result is to continue to adjust, increase the dynamic similarity threshold. The steps are as follows: Feedback to the token sequence library: The merged token will be returned and stored in the token sequence library, which is used for subsequent token compression and replacement, making model training more efficient; Training set generation: Extract compressed and replaced tokens from the token sequence library and build a new training set. These compressed tokens are based on the merged results, thereby improving the quality of the training set and removing redundant information. Input source code model for training: The generated training set is input into the source code model for training. The source code model is based on these new, compressed token learning patterns and structures to perform semantic understanding and prediction. Adjusting the dynamic similarity threshold: After the model is trained, the similarity between tokens is evaluated based on the training results. The model calculates the similarity between tokens. If the model finds sufficient similarity, the dynamic similarity threshold will be adjusted. This threshold is used to control which tokens are considered "similar" and thus decide whether to merge into a new token; Feedback to developers: After training is completed, the model will provide feedback to developers based on the adjusted threshold. If the developer requests further adjustments (such as increasing or decreasing the tolerance for similarity), the dynamic similarity threshold will be appropriately increased or decreased. Increase the dynamic similarity threshold: Based on developer feedback, increase the dynamic similarity threshold (i.e., expand the range of accepted similarities) or lower it. This adjustment will help make the system more lenient or strict in identifying token similarities, thereby further optimizing the merging and compression process.

[0032] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited by the above embodiments. The above embodiments and descriptions are only preferred examples of the present invention and are not intended to limit the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. A dimensionality reduction method for source code security detection large model based on homomorphic merging, characterized by: The following steps are involved: S1. Collect source code data from the development environment and obtain computer performance data; S2. Extract and analyze the data features of the source code data, and convert the source code data into a token series based on the data features according to the analysis results, and assign a position code to each token; S3. Perform redundancy detection on all tokens to extract the same tokens for homomorphic merging. Replace the multiple tokens after homomorphic merging with a new token for representation, and then establish a token sequence library to collect the tokens. S4. Calculate the similarity values ​​of the data features of all tokens in the token sequence library, perform adaptive analysis of the number of tokens based on the complexity of the performance data and source code data, and set a dynamic similarity threshold based on the analysis results; S5. The similarity values ​​between tokens are combined with the dynamic similarity threshold for screening, and then the tokens that meet the load dynamic similarity threshold are homomorphically merged according to the screening results. At the same time, the data dimension reduction analysis is performed on the tokens after homomorphic merger to generate representative feature data; S6: Feed the token processed by S5 back to the token sequence library for compression and replacement, then input the token sequence library into the source code model for training, and adjust the dynamic similarity threshold according to the training results.

2. According to claim 1, a source code security detection large model dimensionality reduction method based on homomorphic merging is characterized by: The S1 collects source code data from different development environments and formats and adjusts the source code data, thereby unifying the format of the source code data.

3. According to claim 1, a source code security detection large model dimensionality reduction method based on homomorphic merging is characterized by: The S1 is connected to the management port of the computer, thereby extracting the performance data of the computer in real time from the management port, thereby monitoring the load status of the computer.

4. According to claim 1, a source code security detection large model dimensionality reduction method based on homomorphic merging is characterized by: The steps of S2 are as follows: S2.

1. Extract features from the source code data to obtain static features and semantic features of the source code data, then aggregate the static features and semantic features into data features, and perform security detection on the source code data based on the data features, and only retain the source code data whose data features pass the security detection; S2.2, convert the source code data retained in S2.1 into a token series with data features, where the token series consists of multiple tokens, each of which represents a basic unit, function name, function block, and process in the source code; S2.

3. Assign a position code to each token so that each token corresponds to a separate position code. The position code is used to represent the application of the token, thereby describing the source code data as a series of token reference codes.

5. According to claim 1, a source code security detection large model dimensionality reduction method based on homomorphic merging is characterized by: The steps of S3 are as follows: S3.

1. Use the redundancy detection algorithm to perform redundancy detection on all tokens to extract the same token, and then homomorphically merge multiple consecutive combinations of repeated tokens to form a new token. S3.

2. Establish a token sequence library, and use the token sequence library to collect the tokens reorganized in S3.1 and the tokens that have not been merged.

6. According to claim 1, a source code security detection large model dimensionality reduction method based on homomorphic merging is characterized by: The steps of S4 are as follows: S4.

1. Extract the data features of all tokens in the token sequence library, and then calculate the similarity values ​​of the data features of all tokens to obtain the similarity values ​​between each token; S4.

2. Combine the source code data corresponding to all tokens in the token sequence library to perform complexity numerical analysis, obtain the complexity value of the token sequence library based on the analysis results, and then combine the performance data with the complexity value to perform token quantity adaptation analysis, obtain the number of tokens that the performance data can adapt to based on the analysis results, and set the number of tokens obtained from the analysis as the dynamic similarity threshold.

7. The method for reducing the dimension of a large model of source code security detection based on homomorphic merging according to claim 6 is characterized in that: In S4.2, when the latest performance data shows that the load state of the computer is overloaded, the latest performance data is substituted into S4.2 to reset the dynamic similarity threshold. Conversely, when the latest performance data shows that the load state of the computer is running normally, monitoring is continued.

8. According to claim 1, a source code security detection large model dimensionality reduction method based on homomorphic merging is characterized by: The steps of S5 are as follows: S5.

1. Filter the similarity values ​​between tokens in combination with the dynamic similarity threshold, so as to form a token merge list that meets the dynamic similarity threshold according to the similarity value composition, and then calculate the average similarity value of the obtained token merge list, and give priority to retaining the token merge list with the highest average similarity value; S5.

2. All tokens are homomorphically merged according to the token merge list retained in S5.

1. At the same time, data features are extracted from the homomorphically merged tokens. The extracted data features are then combined for feature selection and data dimension reduction analysis. According to the analysis results, corresponding representative feature data is generated for the homomorphically merged tokens.

9. The method for reducing the dimension of a large model of source code security detection based on homomorphic merging according to claim 1 is characterized in that: The steps of S6 are as follows: S6.1, feed back the tokens after homomorphic merging in S5.2 to the token sequence library for compression and replacement, so that the token sequence library uses the latest compressed and replaced tokens as the training set; S6.

2. Input the training set into the source code model for training, and adjust the dynamic similarity threshold according to the training results. Then, the training results are fed back to the developer. When the developer's feedback result is to continue adjustment, the dynamic similarity threshold is increased.

Images