Web vulnerability scanning method, system and device and medium

Abstract

The invention discloses a Web vulnerability scanning method, system and device and a medium, and the method comprises the steps: data collection: obtaining a source code and a configuration file of a target Web application, and data and historical vulnerability information of the application during operation; feature extraction: analyzing source codes to extract key function calling, parameter passing and control flow information, and constructing semantic representation of codes; analyzing runtime data, and constructing a comprehensive feature vector in combination with a static code analysis result; model training: matching the extracted features with known vulnerability tags to form a training set and a test set; a deep learning model is selected for training according to problem characteristics, and hyper-parameters are adjusted to optimize model performance; vulnerability detection: executing automatic scanning on the Web application to be detected, and judging whether vulnerabilities exist or not according to the features extracted in real time; and generating a report. According to the method, a comprehensive automatic process from data collection to vulnerability report generation is realized, the scanning efficiency and accuracy are remarkably improved, and the security risk is effectively reduced.

Description

Technical Field

[0001] The present invention belongs to the technical field of network security and relates to a Web vulnerability scanning method, system, equipment and medium. Background Art

[0002] With the rapid development of the Internet and the widespread popularity of Web applications, network security issues have become an important challenge in the digital age. Web applications have become the main target of hacker attacks due to their openness and interactivity. Common Web vulnerabilities, such as SQL injection, cross-site scripting (XSS), file inclusion vulnerabilities, etc., seriously threaten the security of users' personal information, and may also lead to serious consequences such as corporate data leakage and service interruption. Therefore, ensuring the security of Web applications is crucial to protecting user privacy and maintaining corporate operations.

[0003] Currently, Web vulnerability scanning is one of the important means to ensure the security of Web applications. Existing Web vulnerability scanning technologies are mainly divided into two categories: static code analysis and dynamic behavior monitoring. Static analysis technology identifies potential security defects by checking source code or compiled code. This method is particularly effective in the development stage, but its detection effect is often limited due to the accessibility and complexity of the code. Dynamic monitoring technology observes the behavior of the application in the actual operating environment and looks for abnormal patterns. Although it can capture vulnerabilities in actual operation, it also faces problems such as high performance overhead and high false alarm rate.

[0004] In order to improve the accuracy and efficiency of scanning, some advanced scanning tools have introduced machine learning algorithms to identify unknown or deformed attack methods by training models. However, there are still many shortcomings. First of all, the high false positive rate is a prominent problem. Due to the lack of in-depth understanding of the code context, many scanning tools often misjudge harmless code snippets as vulnerabilities, which not only increases the workload of subsequent verification and repair, but also may lead to a decrease in the trust of the security team. Secondly, the problem of missed reports is equally serious. For malicious code that uses the latest attack technology or has been obfuscated, traditional scanning methods are often difficult to effectively identify, leaving security risks.

[0005] In addition, as network attack methods continue to evolve, fixed-rule scanning strategies are difficult to update in a timely manner, and their ability to combat new threats is limited. Scanning tools are not only required to have the ability to be continuously updated, but also to be able to intelligently adapt to the ever-changing network threat environment. Finally, high performance overhead is another issue that cannot be ignored. The deep scanning process may consume a lot of computing resources, especially in high-concurrency scenarios, which may have a serious impact on normal business operations.

[0006] In summary, the existing Web vulnerability scanning technology still has many deficiencies in terms of false positive rate, false negative rate, adaptability and performance overhead, etc. Therefore, the present invention aims to provide a more intelligent and efficient Web vulnerability scanning method to overcome the shortcomings of the existing technology and improve the security of Web applications. Summary of the invention

[0007] The purpose of the present invention is to solve the problem that the Web vulnerability scanning technology in the prior art still has many deficiencies in terms of false alarm rate, missed alarm rate, adaptability and performance overhead, and to provide a Web vulnerability scanning method, system, device and medium.

[0008] In order to achieve the above object, the present invention adopts the following technical solutions: A web vulnerability scanning method comprises the following steps: Data collection: obtaining the target Web application’s source code and configuration files, application runtime data, and historical vulnerability information; Feature extraction: Use natural language processing technology to parse source code, extract key function calls, parameter transfer, and control flow information, and build a semantic representation of the code; analyze runtime data, identify abnormal behavior patterns, and combine static code analysis results to build a comprehensive feature vector; Model training: Pair the extracted features with known vulnerability labels to form training sets and test sets; select deep learning models for training based on the characteristics of the problem, and adjust hyperparameters to optimize model performance; use cross-validation methods to evaluate the performance of the model on different data sets to avoid overfitting; as new vulnerabilities emerge, use incremental learning strategies to update the model so that the model can adapt to new attack patterns; Vulnerability detection: perform automated scanning on the web application to be tested, determine whether there is a vulnerability based on the features extracted in real time, and give a confidence score; establish a feedback mechanism to allow security experts to review suspected vulnerability reports, mark false positives and false negatives, and use them to optimize the model; Report generation: It generates detailed vulnerability reports, including vulnerability location, type, risk level and repair suggestions; it provides an intuitive user interface, displays an overview of the scan results, and allows users to explore the details of specific vulnerabilities in depth. The interface supports search, filtering and sorting functions.

[0009] The data collection specifically includes: Use crawler technology to automatically crawl the source code and configuration files of the target Web application, which contain the basic structure and configuration information of the Web application; collect the data of the Web application at runtime by simulating user requests or monitoring the actual operating environment, and the runtime data includes HTTP requests / responses, database queries, server logs, etc.; obtain historical vulnerability information from a known vulnerability database as the initial data set, and the historical vulnerability information includes the description of the vulnerability, exploit code, and impact range.

[0010] The feature extraction specifically includes: Code parsing and feature extraction: Parsing the source code using an abstract syntax tree, including lexical analysis to decompose the source code into tokens, syntactic analysis to organize the tokens into an abstract syntax tree according to the grammatical rules of the programming language, and traversing the abstract syntax tree to visit each node; During the traversal process, key features of the source code are extracted, including key function calls, parameter passing, and control flow information, which are used to construct a semantic representation of the code; Convert the extracted features into numerical vectors to form code feature vectors; Behavior monitoring and feature construction: Collect runtime data of the target web application, including system logs, network traffic, process information, and memory usage; Preprocess the collected raw data to remove noise and outliers and extract information; Extract features from preprocessed data, including statistics, ratios, and state changes, to reflect system behavior; Analyze the extracted features using statistical analysis methods to identify patterns of normal and abnormal behavior and establish a baseline for normal behavior; Based on the established normal behavior patterns, new runtime data is monitored in real time to determine whether there are abnormal behaviors and convert the monitoring results into behavior feature vectors; Combine the code feature vector and behavior feature vector to construct a comprehensive feature vector for subsequent Web vulnerability detection.

[0011] The model training is specifically as follows: Dataset preparation: Pair features extracted from source code and behavior monitoring with known vulnerability labels to form training and test sets; Model selection: According to the characteristics of the web vulnerability scanning problem, a deep learning model is selected for training. If the feature to be processed is image recognition-related, a convolutional neural network (CNN) model is selected. The CNN model includes an input layer for receiving image data, a convolution layer for extracting local features, an activation function layer for introducing nonlinearity, a pooling layer for downsampling, a fully connected layer for feature integration, and an output layer for classification. If the features related to sequence data are being processed, a recurrent neural network (RNN) or a long short-term memory (LSTM) model is selected. The recurrent neural network (RNN) or the long short-term memory (LSTM) model includes an input layer for receiving sequence data, a recurrent unit or a gating mechanism for capturing time dependencies, a feature extraction layer for extracting useful features from the sequence, a fully connected layer for feature integration, and an output layer for classification or prediction.

[0012] The specific process of training the CNN model is as follows: For the CNN model, the raw image data is received through the input layer; In the convolution layer, the convolution kernel is used to perform sliding convolution on the image to extract local features of the image; Introducing nonlinearity through activation function layers to enhance the expressiveness of the model; In the pooling layer, the feature map is downsampled to reduce the computational complexity; In the fully connected layer, the extracted features are integrated for the final classification task; In the output layer, the classification results are output through the softmax function; The back-propagation algorithm is used to optimize the parameters of the CNN model to minimize the loss function.

[0013] The specific process of RNN / LSTM model training is as follows: For RNN or LSTM models, the raw sequence data is received through the input layer; In recurrent units or gating mechanisms, capturing temporal dependencies in sequences; Extract useful features from sequence data; In the fully connected layer, the extracted features are integrated; In the output layer, the classification results or predicted values ​​are output; Use the back-propagation algorithm to optimize the parameters of the RNN or LSTM model to minimize the loss function.

[0014] The hyperparameters are adjusted to optimize the model performance, specifically: Determine the possible range of values ​​for each hyperparameter; Select a search algorithm to explore the hyperparameter space to optimize model performance; The K-fold cross-validation method is used to evaluate the performance of different hyperparameter combinations on the data set to avoid overfitting. The K-fold cross-validation method includes: Randomly divide the original data set into K subsets of equal size; For each subset, use it as a validation set, and combine the remaining K-1 subsets as training sets. Train the model and test the performance on the validation set. Calculate the performance index of the model on the validation set in each iteration, and repeat the above process K times until each subset has been used as a validation set once; Calculate the average of the model performance indicators across all iterations to evaluate the overall performance of the model; Adjust the model hyperparameters based on the cross-validation results until a hyperparameter combination that meets performance requirements is obtained; After obtaining the optimal hyperparameter combination, final verification is performed on an independent test set.

[0015] A web vulnerability scanning system includes the following modules: The data collection module is used to obtain the source code and configuration files of the target Web application, the application's runtime data, and historical vulnerability information; The feature extraction module uses natural language processing technology to parse source code, extract key function calls, parameter transfer, control flow and other information, and build a semantic representation of the code; analyze runtime data, identify abnormal behavior patterns, and build a comprehensive feature vector based on static code analysis results; The model training module pairs the extracted features with known vulnerability labels to form training sets and test sets; selects deep learning models for training based on the characteristics of the problem, and adjusts hyperparameters to optimize model performance; uses cross-validation methods to evaluate the performance of the model on different data sets to avoid overfitting; and uses incremental learning strategies to update the model as new vulnerabilities emerge, so that the model can adapt to new attack patterns; The vulnerability detection module performs automated scanning on the Web application to be tested, determines whether there are vulnerabilities based on the features extracted in real time, and gives a confidence score; establishes a feedback mechanism to allow security experts to review suspected vulnerability reports, mark false positives and false negatives, and use them to optimize the model; The report generation module compiles detailed vulnerability reports, including vulnerability location, type, risk level and repair suggestions; provides an intuitive user interface to display an overview of scan results and allow users to explore detailed information of specific vulnerabilities in depth. The interface supports search, filtering and sorting functions.

[0016] A device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method as described in any one of the preceding items when executing the computer program.

[0017] A computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the steps of the method described in any one of the preceding items.

[0018] Compared with the prior art, the present invention has the following beneficial effects: A Web vulnerability scanning method proposed in the present invention realizes a comprehensive automated process from data collection to vulnerability report generation. Through data collection, the source code, configuration files, runtime data and historical vulnerability information of the target Web application are efficiently obtained. Subsequently, feature extraction is used, combined with natural language processing technology and runtime data analysis, to construct a comprehensive feature vector. On this basis, the model training adopts a deep learning algorithm, and by adjusting hyperparameters and optimizing model performance, accurate identification of potential vulnerabilities is achieved. An incremental learning strategy is also incorporated to enable the model to be continuously updated as new vulnerabilities are discovered, thereby maintaining a keen perception of new attack patterns. The generated vulnerability report is detailed, including vulnerability location, type, risk level and repair suggestions, and an intuitive and easy-to-use user interface is provided to facilitate users to quickly locate and handle key vulnerabilities. Not only does it significantly improve scanning efficiency and accuracy, it also effectively reduces security risks and reduces the workload of security personnel, showing a wide range of application prospects and important practical value in the field of Web application security protection. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0020] Figure 1 This is a flow chart of the Web vulnerability scanning method of the present invention. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0022] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0023] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0024] The present invention is further described in detail below in conjunction with the accompanying drawings: See also Figure 1 , is a flow chart of the Web vulnerability scanning method of the present invention, comprising the following steps: S1, data collection, obtains the source code and configuration files of the target Web application, the application's runtime data, and historical vulnerability information.

[0025] Use crawler technology to automatically crawl the source code and configuration files of the target Web application, which contain the basic structure and configuration information of the Web application; collect the data of the Web application at runtime by simulating user requests or monitoring the actual operating environment, and the runtime data includes HTTP requests / responses, database queries, server logs, etc.; obtain historical vulnerability information from a known vulnerability database as the initial data set, and the historical vulnerability information includes the description of the vulnerability, exploit code, and impact range.

[0026] S2, feature extraction, uses natural language processing technology to parse source code, extract key function calls, parameter passing, control flow and other information, and build a semantic representation of the code; analyze runtime data, identify abnormal behavior patterns, and combine static code analysis results to build a comprehensive feature vector.

[0027] Specifically include: Code parsing and feature extraction: Parsing the source code using an abstract syntax tree, including lexical analysis to decompose the source code into tokens, syntactic analysis to organize the tokens into an abstract syntax tree according to the grammatical rules of the programming language, and traversing the abstract syntax tree to visit each node; During the traversal process, key features of the source code are extracted, including key function calls, parameter passing, and control flow information, which are used to construct a semantic representation of the code; Convert the extracted features into numerical vectors to form code feature vectors; Behavior monitoring and feature construction: Collect runtime data of the target web application, including system logs, network traffic, process information, and memory usage; Preprocess the collected raw data to remove noise and outliers and extract information; Extract features from preprocessed data, including statistics, ratios, and state changes, to reflect system behavior; Analyze the extracted features using statistical analysis methods to identify patterns of normal and abnormal behavior and establish a baseline for normal behavior; Based on the established normal behavior patterns, new runtime data is monitored in real time to determine whether there are abnormal behaviors and convert the monitoring results into behavior feature vectors; Combine the code feature vector and behavior feature vector to construct a comprehensive feature vector for subsequent Web vulnerability detection.

[0028] S3, model training, pairs the extracted features with known vulnerability labels to form training sets and test sets; selects deep learning models for training based on problem characteristics, and adjusts hyperparameters to optimize model performance; uses cross-validation methods to evaluate the performance of the model on different data sets to avoid overfitting; as new vulnerabilities emerge, uses incremental learning strategies to update the model so that the model can adapt to new attack patterns.

[0029] Model training, specifically: Dataset preparation: Pair features extracted from source code and behavior monitoring with known vulnerability labels to form training and test sets; Model selection: According to the characteristics of the web vulnerability scanning problem, a deep learning model is selected for training. If the feature to be processed is image recognition-related, a convolutional neural network (CNN) model is selected. The CNN model includes an input layer for receiving image data, a convolution layer for extracting local features, an activation function layer for introducing nonlinearity, a pooling layer for downsampling, a fully connected layer for feature integration, and an output layer for classification. If the features related to sequence data are being processed, a recurrent neural network (RNN) or a long short-term memory (LSTM) model is selected. The recurrent neural network (RNN) or the long short-term memory (LSTM) model includes an input layer for receiving sequence data, a recurrent unit or a gating mechanism for capturing time dependencies, a feature extraction layer for extracting useful features from the sequence, a fully connected layer for feature integration, and an output layer for classification or prediction.

[0030] For the CNN model, the raw image data is received through the input layer; In the convolution layer, the convolution kernel is used to perform sliding convolution on the image to extract local features of the image; Introducing nonlinearity through activation function layers to enhance the expressiveness of the model; In the pooling layer, the feature map is downsampled to reduce the computational complexity; In the fully connected layer, the extracted features are integrated for the final classification task; In the output layer, the classification results are output through the softmax function; The back-propagation algorithm is used to optimize the parameters of the CNN model to minimize the loss function.

[0031] For RNN or LSTM models, the raw sequence data is received through the input layer; In recurrent units or gating mechanisms, capturing temporal dependencies in sequences; Extract useful features from sequence data; In the fully connected layer, the extracted features are integrated; In the output layer, the classification results or predicted values ​​are output; Use the back-propagation algorithm to optimize the parameters of the RNN or LSTM model to minimize the loss function.

[0032] Tune hyperparameters to optimize model performance, specifically: Determine the possible range of values ​​for each hyperparameter; Select a search algorithm to explore the hyperparameter space to optimize model performance; The K-fold cross-validation method is used to evaluate the performance of different hyperparameter combinations on the data set to avoid overfitting. The K-fold cross-validation method includes: Randomly divide the original data set into K subsets of equal size; For each subset, use it as a validation set, and combine the remaining K-1 subsets as training sets. Train the model and test the performance on the validation set. Calculate the performance index of the model on the validation set in each iteration, and repeat the above process K times until each subset has been used as a validation set once; Calculate the average of the model performance indicators across all iterations to evaluate the overall performance of the model; Adjust the model hyperparameters based on the cross-validation results until a hyperparameter combination that meets performance requirements is obtained; After obtaining the optimal hyperparameter combination, final verification is performed on an independent test set.

[0033] S4, vulnerability detection, performs automated scanning on the web application under test, determines whether there are vulnerabilities based on the features extracted in real time, and gives a confidence score; establishes a feedback mechanism to allow security experts to review suspected vulnerability reports, mark false positives and false negatives, and use them to optimize the model.

[0034] S5, report generation, compiles a detailed vulnerability report, including vulnerability location, type, risk level and repair suggestions; provides an intuitive user interface, displays an overview of the scan results, and allows users to explore the details of specific vulnerabilities in depth. The interface supports search, filtering and sorting functions. Example

[0035] The deep learning-based Web vulnerability scanning method in the present invention integrates static code analysis, dynamic behavior monitoring and natural language processing technology, and can adaptively learn new attack patterns, thereby significantly reducing false alarm rates and missed alarm rates.

[0036] Detailed description of the specific steps: Step 1: Data Collection Target Web application source code and configuration files: Use crawler technology to automatically crawl the target Web application source code and configuration files. These files contain the basic structure and configuration information of the application, which is the basis for subsequent analysis.

[0037] Runtime data: Collect application runtime data by simulating user requests or monitoring the actual operating environment. This data includes HTTP requests / responses, database queries, server logs, etc., which helps understand the application's behavior patterns.

[0038] Public vulnerability database: Obtain historical vulnerability information from known vulnerability databases (such as CVE, Exploit-DB) as the initial data set for training deep learning models. This information includes vulnerability descriptions, exploit codes, impact scope, etc.

[0039] Step 2: Feature extraction Code parsing: Use natural language processing technology (such as abstract syntax tree AST) to parse source code and extract key function calls, parameter passing, control flow and other information. This information helps build a semantic representation of the code.

[0040] The process of parsing source code using an Abstract Syntax Tree (AST) usually includes the following steps: 1. Lexical analysis: The source code is broken down into a series of tokens, which can be keywords, identifiers, operators, etc. This process is usually completed by a lexical analyzer.

[0041] 2. Syntax analysis: The tokens produced by the lexical analyzer are combined into an abstract syntax tree. The parser organizes these tokens into a hierarchy to represent the structure of the source code according to the grammatical rules of the programming language. Each node represents a grammatical construct, such as an expression, statement, or declaration.

[0042] 3. Traverse the abstract syntax tree: Once the abstract syntax tree is generated, it can be traversed. The purpose of traversal is to visit each node in the tree and perform specific operations on each node. Common traversal methods are depth-first search (DFS) and breadth-first search (BFS).

[0043] 4. Feature extraction: During the traversal process, the features of the source code can be extracted. These features may include structural information of the code, such as the number of loop nesting levels, variable usage, function calls, etc. The extracted features can be customized according to specific application requirements.

[0044] 5. Feature vectorization: Finally, the extracted features are converted into numerical vectors.

[0045] Behavior monitoring: Analyze runtime data and identify abnormal behavior patterns, such as unusual request frequency, abnormal data processing logic, etc. Combined with static code analysis results, a comprehensive feature vector is constructed.

[0046] 1. Data collection: First, you need to collect runtime data of the target system. This includes system logs, network traffic, process information, memory usage, etc. This data can be obtained through various monitoring tools and technical means, such as using logging tools, performance monitoring software, or network packet capture tools.

[0047] 2. Data preprocessing: Clean and process the collected raw data to remove noise and outliers and extract useful information. This may involve operations such as removing irrelevant data items, formatting data, and filling in missing values.

[0048] 3. Feature Engineering: Extract meaningful features from preprocessed data for subsequent analysis. Features can be statistics such as average, maximum, minimum, ratios such as CPU utilization, state changes such as changes in the number of network connections, etc. The purpose of feature engineering is to convert raw data into feature vectors that can reflect system behavior.

[0049] 4. Pattern recognition: Analyze features using statistical analysis methods to identify patterns of normal and abnormal behaviors. Commonly used techniques include cluster analysis, classification algorithms, time series analysis, etc. Models are trained to distinguish different types of behaviors and establish a baseline for normal behavior.

[0050] 5. Anomaly detection: Based on the established normal behavior patterns, new runtime data is monitored in real time to determine whether there is abnormal behavior. If behavior that does not conform to the normal pattern is found, an alarm is triggered or other measures are taken.

[0051] 6. Result interpretation: For the abnormal behavior detected, further analysis and interpretation are required to determine its cause and scope of impact.

[0052] 7. Feedback loop: Update models and benchmarks based on monitoring results to improve the accuracy and efficiency of future monitoring.

[0053] Behavioral monitoring is an ongoing process that requires constant adjustment and improvement.

[0054] Step 3: Model training Dataset preparation: Pair the extracted features with known vulnerability labels (normal / abnormal) to form training and test sets. Ensure the diversity and representativeness of the dataset to improve the generalization ability of the model.

[0055] Model selection: Choose a suitable deep learning model based on the characteristics of the problem, such as convolutional neural network (CNN) for image recognition tasks, recurrent neural network (RNN) or long short-term memory network (LSTM) for sequence data processing.

[0056] The specific analysis process of convolutional neural network (CNN) image recognition is as follows: 1. Input layer: Receives raw image data, which is a two-dimensional matrix or three-dimensional tensor (for color images) consisting of pixel values.

[0057] 2. Convolution layer: The convolution kernel slides over the image and performs convolution operations to extract local features. The convolution kernel can capture features such as texture and edges in the image. This step generates multiple feature maps, each corresponding to a convolution kernel.

[0058] 3. Activation function layer: Introduces nonlinearity. Common activation functions include ReLU, sigmoid, and tanh. These functions can help the network learn more complex features.

[0059] 4. Pooling layer: Reduce the size of the feature map through downsampling operations, reduce computational complexity and improve the generalization ability of the model. Pooling methods include maximum pooling and average pooling.

[0060] 5. Fully connected layer: Integrates the features extracted by the previous layers for the final classification task. The fully connected layer maps the high-dimensional feature vector to the output category space.

[0061] 6. Output layer: Outputs the final classification results, usually by converting the output into a probability distribution through a softmax function for multi-classification tasks.

[0062] 7. Training process: Use the back-propagation algorithm to optimize the parameters of the CNN to minimize the loss function.

[0063] 8. Prediction process: Once the model is trained, it can make predictions on new image data and identify the content in the image.

[0064] Convolutional neural networks automatically extract image features through a hierarchical structure, making it possible to handle complex image recognition tasks.

[0065] Recurrent Neural Networks (RNNs) and Long Short-Term Memory Networks (LSTMs) are key models for processing sequence data and are widely used in natural language processing, time series prediction, speech recognition, and other fields. RNNs capture temporal dependencies in sequences through a cyclic structure, while LSTMs solve the vanishing gradient problem in RNNs through special memory units.

[0066] Recurrent Neural Network (RNN) is a type of neural network used to process sequence data. Its characteristic is that it can capture the temporal dependencies in the sequence. Through the recurrent structure, RNN makes the output of each time step depend not only on the current input, but also on the output of the previous time step.

[0067] Structure of RNN: The core of RNN is a recurrent unit, the state of which is passed between time steps. This enables RNN to remember the information of the previous time step and combine it with the current input to generate the current output.

[0068] Time-step dependency: This time-dependency of RNN makes it very suitable for processing sequence data such as text, speech, time series, etc.

[0069] Limitations: Although RNN performs well in dealing with short-term dependencies, it faces the gradient vanishing and gradient exploding problems as the sequence length increases. This makes it difficult for RNN to capture long-range temporal dependencies.

[0070] Long Short-Term Memory (LSTM) is a variant of RNN that aims to solve the gradient vanishing and gradient exploding problems in RNN. LSTM introduces special memory units that can maintain the validity of information over a longer period of time, thereby capturing long-distance dependencies.

[0071] LSTM structure: LSTM introduces cell state and three gating mechanisms (input gate, forget gate, output gate). Through these gating mechanisms, LSTM can control the inflow, outflow, and retention of information, thereby better managing long-term dependencies.

[0072] Input gate: controls the amount of current input information entering the cell state.

[0073] Forget gate: determines which information in the current cell state needs to be forgotten.

[0074] Output gate: determines what information is output from the cell state.

[0075] Advantages of LSTM: LSTM effectively solves the vanishing gradient problem of RNN through the gating mechanism, allowing the model to maintain good performance when processing long sequence data. This makes LSTM widely used in natural language processing, time series prediction and other fields.

[0076] The specific analysis process is: 1. Input layer: receives raw sequence data, usually a one-dimensional vector representation, such as word sequences, time series data, etc.

[0077] 2. Cyclic unit (for RNN): Through the cyclic structure, the output of each time step depends not only on the current input, but also on the hidden state of the previous time step. This enables RNN to capture the temporal dependencies in the sequence.

[0078] 3. Gating mechanism (for LSTM): LSTM introduces cell state and three gating mechanisms (input gate, forget gate, output gate) to control the flow and retention of information, thereby better managing long-term dependencies.

[0079] 4. Feature extraction: Through recurrent units or gating mechanisms, the model extracts useful features from sequence data, which can be used for subsequent classification or prediction tasks.

[0080] 5. Fully connected layer: Integrate the extracted features for final classification or prediction. The fully connected layer maps the high-dimensional feature vector to the output category space.

[0081] 6. Output layer: Outputs the final classification result or prediction value, usually by converting the output into a probability distribution through a softmax function for multi-classification tasks.

[0082] 7. Training process: Use the back-propagation algorithm to optimize the parameters of the model to minimize the loss function.

[0083] 8. Prediction process: Once the model is trained, it can be used to predict new sequence data, identify the content in the sequence or predict future trends.

[0084] Model training: Perform model training on a high-performance computing platform and adjust hyperparameters to optimize model performance. Use cross-validation methods to evaluate the performance of the model on different data sets to avoid overfitting.

[0085] 1. Determine the range of hyperparameters: First, you need to determine the possible range of values ​​for each hyperparameter based on experience and prior knowledge. For example, the learning rate is usually between 0 and 1, and the number of iterations may be a specific integer range, such as 100 to 1000.

[0086] 2. Select a search algorithm: Next, select a suitable search algorithm to explore the hyperparameter space. Common search algorithms include grid search, random search, Bayesian optimization, and genetic algorithms. These algorithms have their own advantages and disadvantages and need to be selected based on specific tasks and computing resources.

[0087] 3. Perform cross validation: In order to evaluate the performance of different hyperparameter combinations, K-fold cross validation is usually used. In each validation, the data set is divided into K subsets, one of which is used as a validation set in turn, and the remaining K-1 subsets are used as training sets, and so on, until each subset is used as a validation set once.

[0088] 4. Evaluate model performance: During each validation process, the model is trained using the current hyperparameter combination and its performance is evaluated on the validation set. This can be done by calculating metrics such as loss function, accuracy, recall, etc.

[0089] 5. Update hyperparameters: Update the values ​​of hyperparameters based on the performance of the model on the validation set.

[0090] 6. Repeat iteration: Repeat the above process until a hyperparameter combination that meets performance requirements is found.

[0091] 7. Final verification: Once the optimal hyperparameter combination is found, final verification is required on an independent test set to ensure that the model has good generalization ability.

[0092] The specific evaluation process of the model was evaluated using the cross-validation method.

[0093] 1. Data partitioning: Randomly divide the original data set into K subsets of equal size. Each subset is called a fold.

[0094] 2. Cross-validation iteration: For each fold, use it as a validation set (hold-out set), and the remaining K-1 subsets are combined as training sets. Then, train the model on the training set and test the performance of the model on the validation set.

[0095] 3. Performance evaluation: Calculate the performance indicators of the model on the validation set in each iteration, such as accuracy, recall, etc. These indicators reflect the predictive ability of the model in that iteration.

[0096] 4. Average performance calculation: Repeat the above process K times, each time selecting a different fold as the validation set, until each fold has been used as a validation set once. Finally, calculate the average of the model performance indicators in all iterations, which represents the overall performance of the model.

[0097] 5. Result analysis: Based on the calculated average performance index, the generalization ability of the model can be evaluated. If the performance of the model on all folds is relatively stable, it means that the model has a high generalization ability; otherwise, there may be problems of overfitting or underfitting.

[0098] 6. Parameter adjustment: Based on the results of cross-validation, the model's hyperparameters may need to be adjusted to optimize the model's performance. This process may involve multiple iterations until the best hyperparameter combination is found.

[0099] 7. Final validation: After determining the optimal hyperparameters, a final validation is performed on an independent test set to ensure that the model has good generalization ability.

[0100] Incremental learning: As new vulnerabilities emerge, the model is updated regularly and an incremental learning strategy is adopted to enable the model to adapt to new attack patterns.

[0101] Step 4: Vulnerability Detection Real-time scanning: Perform automated scanning on the web application to be tested. The model determines whether there is a vulnerability based on the features extracted in real time and gives a confidence score. This process should be as fast as possible and should not affect the normal operation of the application.

[0102] False positive and false negative management: Establish a feedback mechanism to allow security experts to review suspected vulnerability reports and mark false positives and false negatives for further model optimization.

[0103] Step 5: Report generation Detailed report: The system automatically generates a detailed vulnerability report, including vulnerability location, type, risk level, and fix recommendations. The report should be easy to understand and contain enough technical details to guide developers in fixing the problem.

[0104] Visual interface: Provide an intuitive user interface that displays an overview of scan results and allows users to drill down to explore detailed information about specific vulnerabilities. The interface should support multiple interactive methods, such as searching, filtering, and sorting.

[0105] Through the above steps, the present invention realizes an efficient and accurate Web vulnerability scanning system, which can significantly improve the security of Web applications.

[0106] An embodiment of the present invention is a web vulnerability scanning system, comprising the following modules: The data collection module is used to obtain the source code and configuration files of the target Web application, the application's runtime data, and historical vulnerability information; The feature extraction module uses natural language processing technology to parse source code, extract key function calls, parameter transfer, control flow and other information, and build a semantic representation of the code; analyze runtime data, identify abnormal behavior patterns, and build a comprehensive feature vector based on static code analysis results; The model training module pairs the extracted features with known vulnerability labels to form training sets and test sets; selects deep learning models for training based on the characteristics of the problem, and adjusts hyperparameters to optimize model performance; uses cross-validation methods to evaluate the performance of the model on different data sets to avoid overfitting; and uses incremental learning strategies to update the model as new vulnerabilities emerge, so that the model can adapt to new attack patterns; The vulnerability detection module performs automated scanning on the Web application to be tested, determines whether there are vulnerabilities based on the features extracted in real time, and gives a confidence score; establishes a feedback mechanism to allow security experts to review suspected vulnerability reports, mark false positives and false negatives, and use them to optimize the model; The report generation module compiles detailed vulnerability reports, including vulnerability location, type, risk level and repair suggestions; provides an intuitive user interface to display an overview of scan results and allow users to explore detailed information of specific vulnerabilities in depth. The interface supports search, filtering and sorting functions.

[0107] An embodiment of the present invention provides a terminal device. The terminal device of this embodiment includes: a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps in the above-mentioned method embodiments are implemented. Alternatively, when the processor executes the computer program, the functions of the modules / units in the above-mentioned device embodiments are implemented.

[0108] The computer program may be divided into one or more modules / units, and the one or more modules / units are stored in the memory and executed by the processor to accomplish the present invention.

[0109] The device / terminal equipment may be a computing device such as a desktop computer, a notebook, a palm computer, a cloud server, etc. The device / terminal equipment may include, but is not limited to, a processor and a memory.

[0110] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0111] The memory may be used to store the computer programs and / or modules, and the processor implements various functions of the apparatus / terminal device by running or executing the computer programs and / or modules stored in the memory and calling the data stored in the memory.

[0112] If the module / unit integrated in the device / terminal equipment is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, the steps of the above-mentioned various method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device that can carry the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0113] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A Web vulnerability scanning method, characterized in that: The following steps are involved: Data collection: obtaining the target Web application’s source code and configuration files, application runtime data, and historical vulnerability information; Feature extraction: Use natural language processing technology to parse source code, extract key function calls, parameter transfer, and control flow information, and build a semantic representation of the code; analyze runtime data, identify abnormal behavior patterns, and combine static code analysis results to build a comprehensive feature vector; Model training: Pair the extracted features with known vulnerability labels to form training sets and test sets; select deep learning models for training based on the characteristics of the problem, and adjust hyperparameters to optimize model performance; use cross-validation methods to evaluate the performance of the model on different data sets to avoid overfitting; as new vulnerabilities emerge, use incremental learning strategies to update the model so that the model can adapt to new attack patterns; Vulnerability detection: perform automated scanning on the web application to be tested, determine whether there are vulnerabilities based on the features extracted in real time, and give a confidence score; Establish a feedback mechanism to allow security experts to review suspected vulnerability reports and mark false positives and false negatives for model optimization; Report generation: It generates detailed vulnerability reports, including vulnerability location, type, risk level and repair suggestions; it provides an intuitive user interface, displays an overview of the scan results, and allows users to explore the details of specific vulnerabilities in depth. The interface supports search, filtering and sorting functions.

2. A Web vulnerability scanning method as claimed in claim 1, characterized in that: The data collection specifically includes: Use crawler technology to automatically crawl the source code and configuration files of the target Web application, which contain the basic structure and configuration information of the Web application; collect the data of the Web application at runtime by simulating user requests or monitoring the actual operating environment, and the runtime data includes HTTP requests / responses, database queries, server logs, etc.; obtain historical vulnerability information from a known vulnerability database as the initial data set, and the historical vulnerability information includes the description of the vulnerability, exploit code, and impact range.

3. A Web vulnerability scanning method as claimed in claim 1, characterized in that: The feature extraction specifically includes: Code parsing and feature extraction: Parsing the source code using an abstract syntax tree, including lexical analysis to decompose the source code into tokens, syntactic analysis to organize the tokens into an abstract syntax tree according to the grammatical rules of the programming language, and traversing the abstract syntax tree to visit each node; During the traversal process, key features of the source code are extracted, including key function calls, parameter passing, and control flow information, which are used to construct a semantic representation of the code; Convert the extracted features into numerical vectors to form code feature vectors; Behavior monitoring and feature construction: Collect runtime data of the target web application, including system logs, network traffic, process information, and memory usage; Preprocess the collected raw data to remove noise and outliers and extract information; Extract features from preprocessed data, including statistics, ratios, and state changes, to reflect system behavior; Analyze the extracted features using statistical analysis methods to identify patterns of normal and abnormal behavior and establish a baseline for normal behavior; Based on the established normal behavior patterns, new runtime data is monitored in real time to determine whether there are abnormal behaviors and convert the monitoring results into behavior feature vectors; Combine the code feature vector and behavior feature vector to construct a comprehensive feature vector for subsequent Web vulnerability detection.

4. A Web vulnerability scanning method as claimed in claim 1, characterized in that: The model training is specifically as follows: Dataset preparation: Pair features extracted from source code and behavior monitoring with known vulnerability labels to form training and test sets; Model selection: According to the characteristics of the web vulnerability scanning problem, a deep learning model is selected for training. If the feature to be processed is image recognition-related, a convolutional neural network (CNN) model is selected. The CNN model includes an input layer for receiving image data, a convolution layer for extracting local features, an activation function layer for introducing nonlinearity, a pooling layer for downsampling, a fully connected layer for feature integration, and an output layer for classification. If the features related to sequence data are being processed, a recurrent neural network (RNN) or a long short-term memory (LSTM) model is selected. The recurrent neural network (RNN) or the long short-term memory (LSTM) model includes an input layer for receiving sequence data, a recurrent unit or a gating mechanism for capturing time dependencies, a feature extraction layer for extracting useful features from the sequence, a fully connected layer for feature integration, and an output layer for classification or prediction.

5. A Web vulnerability scanning method as claimed in claim 4, characterized in that: The specific process of training the CNN model is as follows: For the CNN model, the raw image data is received through the input layer; In the convolution layer, the convolution kernel is used to perform sliding convolution on the image to extract local features of the image; Introducing nonlinearity through activation function layers to enhance the expressiveness of the model; In the pooling layer, the feature map is downsampled to reduce the computational complexity; In the fully connected layer, the extracted features are integrated for the final classification task; In the output layer, the classification results are output through the softmax function; The back-propagation algorithm is used to optimize the parameters of the CNN model to minimize the loss function.

6. A Web vulnerability scanning method as claimed in claim 4, characterized in that: The specific process of RNN / LSTM model training is as follows: For RNN or LSTM models, the raw sequence data is received through the input layer; In recurrent units or gating mechanisms, capturing temporal dependencies in sequences; Extract useful features from sequence data; In the fully connected layer, the extracted features are integrated; In the output layer, the classification results or predicted values ​​are output; Use the back-propagation algorithm to optimize the parameters of the RNN or LSTM model to minimize the loss function.

7. A Web vulnerability scanning method as claimed in claim 1, characterized in that: The hyperparameters are adjusted to optimize the model performance, specifically: Determine the possible range of values ​​for each hyperparameter; Select a search algorithm to explore the hyperparameter space to optimize model performance; The K-fold cross-validation method is used to evaluate the performance of different hyperparameter combinations on the data set to avoid overfitting. The K-fold cross-validation method includes: Randomly divide the original data set into K subsets of equal size; For each subset, use it as a validation set, and combine the remaining K-1 subsets as training sets. Train the model and test the performance on the validation set. Calculate the performance index of the model on the validation set in each iteration, and repeat the above process K times until each subset has been used as a validation set once; Calculate the average of the model performance indicators across all iterations to evaluate the overall performance of the model; Adjust the model hyperparameters based on the cross-validation results until a hyperparameter combination that meets performance requirements is obtained; After obtaining the optimal hyperparameter combination, final verification is performed on an independent test set.

8. A Web vulnerability scanning system, characterized in that: Includes the following modules: The data collection module is used to obtain the source code and configuration files of the target Web application, the application's runtime data, and historical vulnerability information; The feature extraction module uses natural language processing technology to parse source code, extract key function calls, parameter transfer, control flow and other information, and build a semantic representation of the code; analyze runtime data, identify abnormal behavior patterns, and build a comprehensive feature vector based on static code analysis results; The model training module pairs the extracted features with known vulnerability labels to form training sets and test sets; selects deep learning models for training based on the characteristics of the problem, and adjusts hyperparameters to optimize model performance; uses cross-validation methods to evaluate the performance of the model on different data sets to avoid overfitting; and uses incremental learning strategies to update the model as new vulnerabilities emerge, so that the model can adapt to new attack patterns; The vulnerability detection module performs automated scanning on the Web application to be tested, determines whether there are vulnerabilities based on the features extracted in real time, and gives a confidence score; Establish a feedback mechanism to allow security experts to review suspected vulnerability reports and mark false positives and false negatives for model optimization; The report generation module compiles detailed vulnerability reports, including vulnerability location, type, risk level and repair suggestions; provides an intuitive user interface to display an overview of scan results and allow users to explore detailed information of specific vulnerabilities in depth. The interface supports search, filtering and sorting functions.

9. A device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images