A method and system for monitoring open source component vulnerability traffic

Abstract

The present invention proposes a method and system for monitoring vulnerability traffic of open source components, which relates to the technical field of vulnerability traffic monitoring. The method collects and processes data of open source components, obtains component collection and processing data, identifies the open source components, and trains component recognition models to obtain specific recognition data; calculates the recognition accuracy of the model, determines whether to perform incremental training, and obtains incremental update recognition data through incremental training; performs recognition and matching of specific open source components to obtain specific matching data, calculates the specific risk coefficient of the specific open source components, and then obtains threat alarm data; compares the update recognition data with a preset update recognition threshold, and compares the threat alarm data with a preset alarm threshold, adjusts the data collection frequency, and obtains frequency adjustment data. The present invention realizes real-time detection and automatic protection of open source component vulnerabilities, and significantly improves the security and reliability of the system.

Description

Technical Field

[0001] The present invention proposes a method and system for monitoring open source component vulnerability traffic, and relates to the technical field of vulnerability traffic monitoring, and specifically to the technical field of open source component vulnerability traffic monitoring. Background Art

[0002] In recent years, software development has become increasingly dependent on open source components, which has significantly improved development efficiency and code reusability. However, the widespread use of open source components has also brought new security challenges. The number of vulnerabilities in open source components is huge and growing rapidly, and their security risks have become one of the main threats to software system security. Traditional security measures, such as regular security scanning, manual vulnerability repair, and dependency management, often seem powerless in the face of rapidly iterative open source components and increasingly complex attack methods. These methods usually rely on static code analysis or regular scanning, and have problems such as detection lag, high false alarm rate, and difficulty in dealing with zero-day vulnerabilities. In addition, many security scanning tools have difficulty effectively identifying open source components that have been maliciously tampered with or hidden, as well as attacks conducted through network traffic.

[0003] Existing technologies mainly focus on the following aspects: static code analysis tools can detect security vulnerabilities in code, but they cannot detect runtime vulnerabilities or attacks against deployed systems; vulnerability databases, such as NVD and CVE, provide information on known vulnerabilities, but require manual matching and analysis, which is inefficient and easy to miss; intrusion detection systems (IDS) and intrusion prevention systems (IPS) can detect network attacks, but they usually lack a deep understanding of the vulnerabilities of specific open source components, resulting in high false positives and high false negatives. These methods are usually independent and lack effective integration and collaborative working mechanisms. Summary of the invention

[0004] The present invention provides a method and system for monitoring open source component vulnerability traffic to solve the above problems: The present invention provides a method and system for monitoring open source component vulnerability traffic, the method comprising: S1. Collect and process data of open source components through network operation traffic data, obtain component collection and processing data, obtain feature codes and software packages to identify open source components, and train component identification models to obtain specific identification data; S2. Calculate the model recognition accuracy, determine whether to perform incremental training of the component recognition model, obtain the incremental recognition model and update the recognition data through incremental training; S3, matching all identification data with a preset vulnerability database for specific open source components, obtaining specific matching data, calculating a specific risk factor for the specific open source component, and then obtaining threat warning data for the specific open source component with a threat warning label; S4. Compare the update identification data with a preset update identification threshold, and compare the threat warning data with a preset warning threshold, and adjust the data collection frequency according to the obtained threshold comparison information to obtain frequency adjustment data.

[0005] Furthermore, the S1 includes: Acquire network operation flow data, and collect data of open source components on the network operation flow data according to the preset open source component type information to obtain component collection data; Preprocessing the component collected data to obtain component collected and processed data; Obtaining feature codes of preset open source components and software package information in component collection and processing data; Identify the open source component according to the feature code and the software package information to obtain component identification data; Training a deep learning model according to the component identification data to obtain a component identification model; The specific open source component is identified according to the component identification model to obtain specific identification data.

[0006] Further, the S2 includes: Obtain all specific identification data, and calculate the model identification accuracy rate based on the specific identification data; Performing incremental training determination on the component recognition model according to the recognition accuracy of the model; Acquire incremental data of component identification data, and perform incremental training on the component identification model through the incremental data to obtain an incremental identification model; Identify the incremental open source components according to the incremental identification model to obtain update identification data; In a preset time period, the model recognition accuracy is calculated based on the updated recognition data until the incremental training is determined to be unnecessary.

[0007] Further, the S3 includes: Obtain a preset vulnerability database, and perform identification and matching of a specific open source component with the specific identification data or the updated identification data and the preset vulnerability database to obtain specific matching data; Calculate the specific risk factor of a specific open source component through specific matching data; Comparing the specific risk factor with a preset risk threshold to obtain risk comparison information; Performing vulnerability threat determination on the specific open source component according to the risk comparison information to obtain vulnerability threat determination information; Performing threat warning marking on the specific open source component according to the vulnerability threat determination information; Threat alerts are issued for vulnerabilities in specific open source components with threat alert annotations to obtain threat alert data.

[0008] Further, the S4 includes: When the update identification data is greater than or equal to the preset update identification threshold, and the threat warning data is greater than or equal to the preset warning threshold, the data collection frequency of the open source component is increased; When the update identification data is less than the preset update identification threshold and the threat warning data is less than the preset warning threshold, the data collection frequency of the open source component is reduced; When the update identification data is greater than or equal to the preset update identification threshold, and the threat warning data is less than the preset warning threshold, the data collection frequency of the open source component is maintained; When the update identification data is less than a preset update identification threshold and the threat warning data is greater than or equal to the preset warning threshold, the data collection frequency of the open source component is increased or maintained.

[0009] Furthermore, the system comprises: The collection and identification training module is used to collect and process the data of open source components through network operation traffic data, obtain component collection and processing data, obtain feature codes and software packages to identify open source components, and train component identification models to obtain specific identification data; The model incremental training module is used to calculate the model recognition accuracy, determine whether to perform incremental training of the component recognition model, obtain the incremental recognition model through incremental training, and update the recognition data; The matching risk warning module is used to match all identification data with the preset vulnerability database for specific open source components, obtain specific matching data, calculate the specific risk coefficient of the specific open source component, and then obtain the threat warning data of the specific open source component with threat warning annotation; The dual comparison and adjustment module is used to compare the update identification data with the preset update identification threshold, and to compare the threat alarm data with the preset alarm threshold, and to adjust the data collection frequency according to the obtained threshold comparison information to obtain frequency adjustment data.

[0010] Furthermore, the acquisition and recognition training module includes: Acquire network operation flow data, and collect data of open source components on the network operation flow data according to the preset open source component type information to obtain component collection data; Preprocessing the component collected data to obtain component collected and processed data; Obtaining feature codes of preset open source components and software package information in component collection and processing data; Identify the open source component according to the feature code and the software package information to obtain component identification data; Training a deep learning model according to the component identification data to obtain a component identification model; The specific open source component is identified according to the component identification model to obtain specific identification data.

[0011] Furthermore, the model incremental training module includes: Obtain all specific identification data, and calculate the model identification accuracy rate based on the specific identification data; Performing incremental training determination on the component recognition model according to the recognition accuracy of the model; Acquire incremental data of component identification data, and perform incremental training on the component identification model through the incremental data to obtain an incremental identification model; Identify the incremental open source components according to the incremental identification model to obtain update identification data; In a preset time period, the model recognition accuracy is calculated based on the updated recognition data until the incremental training is determined to be unnecessary.

[0012] Furthermore, the matching risk warning module includes: Obtain a preset vulnerability database, and perform identification and matching of a specific open source component with the specific identification data or the updated identification data and the preset vulnerability database to obtain specific matching data; Calculate the specific risk factor of a specific open source component through specific matching data; Comparing the specific risk factor with a preset risk threshold to obtain risk comparison information; Performing vulnerability threat determination on the specific open source component according to the risk comparison information to obtain vulnerability threat determination information; Performing threat warning marking on the specific open source component according to the vulnerability threat determination information; Threat alerts are issued for vulnerabilities in specific open source components with threat alert annotations to obtain threat alert data.

[0013] Furthermore, the dual comparison adjustment module includes: When the update identification data is greater than or equal to the preset update identification threshold, and the threat warning data is greater than or equal to the preset warning threshold, the data collection frequency of the open source component is increased; When the update identification data is less than the preset update identification threshold and the threat warning data is less than the preset warning threshold, the data collection frequency of the open source component is reduced; When the update identification data is greater than or equal to the preset update identification threshold, and the threat warning data is less than the preset warning threshold, the data collection frequency of the open source component is maintained; When the update identification data is less than a preset update identification threshold and the threat warning data is greater than or equal to the preset warning threshold, the data collection frequency of the open source component is increased or maintained.

[0014] Beneficial effects of the present invention: by real-time monitoring of network traffic, combined with known vulnerability databases and machine learning methods, malicious activities related to open source component vulnerabilities can be quickly and accurately identified, and corresponding protection strategies can be automatically generated, thereby effectively improving the security and reliability of the software system. The system makes up for the deficiencies of the existing technology in terms of real-time, automation and pertinence, and provides a new method for ensuring the security of the software system. By combining the use of feature codes and software package information, as well as the training of deep learning models, the recognition accuracy of open source components can be significantly improved. By matching and risk assessment with a preset vulnerability database, potential threats in the system can be monitored in real time, and vulnerabilities can be discovered and handled in a timely manner. By dynamically adjusting the data acquisition frequency according to the threshold comparison information, while ensuring the monitoring effect, resource usage can be optimized and unnecessary computing and storage overhead can be reduced. The entire monitoring process is highly automated, reducing the need for manual intervention. At the same time, through the training and use of deep learning models, intelligent monitoring and response are achieved. Through real-time monitoring and threat alerts, security risks in the system can be discovered and handled in a timely manner, enhancing the overall security of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 The figure is a schematic diagram of a method for monitoring open source component vulnerability traffic. DETAILED DESCRIPTION

[0016] The preferred embodiments of the present invention are described below in conjunction with the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0017] In one embodiment of the present invention, a method and system for monitoring open source component vulnerability traffic is proposed, the method comprising: S1. Collect and process data of open source components through network operation traffic data, obtain component collection and processing data, obtain feature codes and software packages to identify open source components, and train component identification models to obtain specific identification data; S2. Calculate the model recognition accuracy, determine whether to perform incremental training of the component recognition model, obtain the incremental recognition model and update the recognition data through incremental training; S3, matching all identification data with a preset vulnerability database for specific open source components, obtaining specific matching data, calculating a specific risk factor for the specific open source component, and then obtaining threat warning data for the specific open source component with a threat warning label; S4, compare the update identification data with the preset update identification threshold, and compare the threat warning data with the preset warning threshold, and adjust the data collection frequency according to the obtained threshold comparison information to obtain frequency adjustment data, such as Figure 1 shown.

[0018] The working principle of the above technical solution is: collect relevant information through network operation traffic data, which includes the usage of software components, communication mode, etc. Process these raw data to extract the collected and processed data that can be used for identification, such as component name, version, call frequency, etc. Use feature codes and software package information to identify open source components in the traffic. Feature codes can be code snippets or identifiers unique to components, and software package information contains metadata and dependencies of components. Train component recognition models through historical recognition data, and use deep learning methods to improve recognition accuracy. This model can learn the characteristics of components, so as to more accurately identify components in traffic. The deep learning designed in this application is only a method for obtaining historical recognition rules based on historical recognition information. Calculate the model recognition accuracy and evaluate the performance of the current model. If the accuracy is lower than the preset threshold, perform incremental training. Incremental training allows the model to improve its performance through new data without training from scratch, thereby obtaining an updated recognition model and recognition data. Match the identified components with the preset vulnerability database to find out whether there are known vulnerabilities. For successfully matched components, their specific risk coefficient is calculated, which takes into account factors such as the severity of the vulnerability, the popularity of the component, and the importance of the component in the system.

[0019] Based on the risk factor, components with potential threats are marked with threat alerts. Updated identification data is compared with the preset update identification threshold to evaluate the adequacy of current data collection. Threat alert data is compared with the preset alert threshold to understand the urgency of the current threat. Based on these threshold comparison information, the data collection frequency is dynamically adjusted. Based on the comparison relationship between the identification data and alert data and the threshold, the data collection frequency is increased to obtain more detailed information, or the frequency is reduced to reduce resource consumption.

[0020] Specific steps may also include: Collect network traffic data packets from network devices (e.g., switches, firewalls) or servers. Supports multiple collection methods, such as mirror port, SPAN port, network packet capture tools, etc. The collected data packets contain information such as source IP, destination IP, port, protocol, data load, etc.

[0021] Pre-process the collected data packets, for example, remove redundant information, clean data, analyze protocols, etc. For HTTP / HTTPS traffic, extract key information such as request method, URL, request header, request body, etc.

[0022] Based on the preprocessed data, features that may be related to known open source component vulnerabilities are extracted. For example, specific parameters in the URL, specific fields in the request header, specific character sequences in the request body, etc. These features are matched with the features in the vulnerability feature library.

[0023] The extracted features are matched according to the rules in the vulnerability feature library. The rule library contains feature descriptions of various known open source component vulnerabilities, such as CVE numbers, vulnerability descriptions, attack vectors, feature expressions, etc. The matching rules can be regular expressions, signature-based matching, or machine learning models, etc.

[0024] Based on the matching results, the risk is assessed. Assessment factors include vulnerability severity, attack frequency, target system importance, etc. The assessment results are used to determine the priority of the protection strategy.

[0025] Take appropriate protective measures based on the risk assessment results. For example, traffic blocking and alarm notification can be performed. Protective measures can be rule-based or policy-based.

[0026] Record all audit events, including attack attempts, protection measures, risk assessment results, etc. Audit data can be used for security analysis, report generation, security policy optimization, etc.

[0027] The technical effect of the above technical solution is: through the combined use of feature codes and software package information, as well as the training of deep learning models, the recognition accuracy of open source components can be significantly improved. Through matching with the preset vulnerability database and risk assessment, potential threats in the system can be monitored in real time, and vulnerabilities can be discovered and handled in a timely manner. By dynamically adjusting the data collection frequency according to the threshold comparison information, it is possible to optimize resource usage and reduce unnecessary computing and storage overhead while ensuring the monitoring effect. The entire monitoring process is highly automated, reducing the need for manual intervention. At the same time, through the training and use of deep learning models, intelligent monitoring and response are achieved. Through real-time monitoring and threat alerts, security risks in the system can be discovered and handled in a timely manner, enhancing the overall security of the system.

[0028] In one embodiment of the present invention, the S1 includes: Acquire network operation flow data, and collect data of open source components on the network operation flow data according to the preset open source component type information to obtain component collection data; Preprocessing the component collected data to obtain component collected and processed data; Obtaining feature codes of preset open source components and software package information in component collection and processing data; Identify the open source component according to the feature code and the software package information to obtain component identification data; Training a deep learning model according to the component identification data to obtain a component identification model; The specific open source component is identified according to the component identification model to obtain specific identification data.

[0029] The working principle of the above technical solution is: real-time traffic data is captured from the network environment. This data usually contains a large amount of network communication information, including the content, source and destination, timestamp, etc. of the data packet. Using the preset open source component type information, the network operation traffic data is screened and analyzed to identify and extract data related to these open source components. This process involves deep analysis of the traffic data to identify specific protocols, port numbers, data formats, etc., so as to determine which traffic is related to specific open source components. The collected component data is pre-processed such as cleaning, formatting and normalization to improve the quality and availability of the data. Preprocessing may include removing redundant data, filling missing values, converting data types, etc. The signature is the key information for identifying a specific open source component, usually a set of unique code snippets or identifiers. The package information contains the metadata of the component, such as version number, dependencies, etc. This information can be obtained from the official documentation, code library or security database of the open source component. Combining the signature and package information, a deep learning method is used to identify the pre-processed component collection data, aiming to determine which parts of the traffic data include specific open source component information. Using the identified component data as training samples, a deep learning model is trained. This model can learn the feature representation of components and is used to automatically identify and classify new traffic data. The selection and training process of deep learning models involves a variety of methods and algorithms, such as convolutional neural networks (CNNs) and recurrent neural networks (RNNs). Use the trained component identification model to identify specific open source components. This process can be online identification of real-time traffic data or offline analysis of historical data. The identification results can be used in a variety of scenarios such as security monitoring, vulnerability scanning, and risk assessment.

[0030] The technical effect of the above technical solution is: by combining the signature code and software package information for identification, and using the deep learning model for training, the accuracy of open source component identification can be significantly improved. The deep learning model can learn complex feature representations and adapt to the identification needs in different scenarios. The entire identification process is highly automated, reducing the need for manual intervention. The use of deep learning models makes the identification process more intelligent, and can adaptively adjust the identification strategy to cope with different traffic data and component types. The method supports online identification of real-time traffic data, and can promptly detect and handle potential threats. At the same time, due to the good scalability of the deep learning model, it can easily adapt to new component types and signature codes, thereby maintaining the continuous update of the identification capability. By accurately identifying open source components and monitoring their usage, potential security threats can be promptly discovered and responded to. This is of great significance for protecting the integrity, availability and confidentiality of the system.

[0031] In one embodiment of the present invention, S2 includes: Obtain all specific identification data, and calculate the model identification accuracy rate based on the specific identification data; The calculation formula of the model recognition accuracy is:

[0032] Among them, ZU is the recognition accuracy of the model, TP is the number of correct recognitions in all recognition data, YL is the total number of initial feature recognition data, GL is the total number of updated recognition data, and GL may be 0; Performing incremental training determination on the component recognition model according to the recognition accuracy of the model; Acquire incremental data of component identification data, and perform incremental training on the component identification model through the incremental data to obtain an incremental identification model; Identify the incremental open source components according to the incremental identification model to obtain update identification data; In a preset time period, the model recognition accuracy is calculated based on the updated recognition data until the incremental training is determined to be unnecessary.

[0033] The working principle of the above technical solution is as follows: collect all the identification data for specific open source components from the previous steps. These data are a direct reflection of the model recognition results, including the model's recognition of each component. Calculate the model recognition accuracy. This accuracy rate not only takes into account the number of initial recognition data, but also takes into account the number of updated recognition data, which can adapt to the calculation of the model recognition accuracy after the model is updated; it reflects the performance of the model on the current data set. According to the calculated model recognition accuracy, it is compared with the preset accuracy threshold. If the accuracy rate is lower than the threshold, it means that the performance of the model needs to be improved, so incremental training is required. Incremental training is a method of using new data to optimize model performance based on an existing model. Once it is decided to perform incremental training, new component recognition data needs to be collected as incremental data. Use these incremental data to train the existing component recognition model to obtain an updated incremental recognition model. Use the incremental recognition model to identify the new incremental open source component and obtain updated recognition data. Calculate the model recognition accuracy again within a preset time period to evaluate the effect of incremental training. Repeat the above steps, i.e., collect incremental data, perform incremental training, and calculate accuracy, until the model recognition accuracy reaches or exceeds the preset threshold, at which point it is determined that no further incremental training is required. The threshold can be adjusted as the data increases.

[0034] The technical effect of the above technical solution is that through incremental training, the model can continuously adapt to new data and scenarios, thereby improving its recognition accuracy. The effectiveness and reliability of the model in practical applications can be maintained. Compared with training the entire model from scratch, incremental training only requires fine-tuning the model, so it can significantly reduce training time and computing resource consumption.

[0035] As incremental data is continuously added and trained, the model can learn more features and patterns, thereby improving its generalization ability. This means that the model can better handle unknown or unseen data. By continuously iterating training and optimizing the model, the system can adapt to changes in the network environment and new threats more quickly. It can enhance the system's adaptability and robustness and improve overall security. The incremental training process can be combined with actual monitoring to achieve real-time updating and monitoring of the model. This means that the system can detect and handle new threats in a timely manner, improving response speed and accuracy.

[0036] In one embodiment of the present invention, S3 includes: Obtain a preset vulnerability database, and perform identification and matching of a specific open source component with the specific identification data or the updated identification data and the preset vulnerability database to obtain specific matching data; Calculate the specific risk factor of a specific open source component through specific matching data; The calculation formula of the specific risk factor is:

[0037] Among them, TD is the specific risk factor, PC is the number of successful matches between the specific identification data and the preset vulnerability database, PZ is the total number of matches between the specific identification data and the preset vulnerability database, and Q k is the preset weight value of the abnormal type of the preset vulnerability data corresponding to the kth successfully matched specific identification data. H is used to reflect the inverse relationship of the update frequency of the open source component corresponding to the specific identification data, that is, the more frequently the component is updated, the lower its risk factor is due to this factor; Comparing the specific risk factor with a preset risk threshold to obtain risk comparison information; Performing vulnerability threat determination on the specific open source component according to the risk comparison information to obtain vulnerability threat determination information; Performing threat warning marking on the specific open source component according to the vulnerability threat determination information; Threat alerts are issued for vulnerabilities in specific open source components with threat alert annotations to obtain threat alert data.

[0038] The working principle of the above technical solution is as follows: obtain a preset vulnerability database, which contains vulnerability information of various known open source components, including vulnerability description, impact scope, repair suggestions, etc. This database is the basis for subsequent matching and risk assessment. Match specific identification data (or update identification data) with the preset vulnerability database. This process aims to identify which open source components have known vulnerabilities. Matching is based on information such as component name, version, and feature code. For the open source components identified with vulnerabilities, calculate their risk coefficients. Compare the calculated specific risk coefficients with the preset risk threshold. If the risk coefficient exceeds the threshold, it is considered that the open source component has a high vulnerability threat and needs further processing. For the open source components determined to have vulnerability threats, the system performs threat alarm annotations. This usually means marking these components in the system's management interface or log so that administrators can quickly identify and handle them. Threat alarms are issued for specific open source components with threat alarm annotations. This may include sending alarm notifications, generating alarm reports, triggering automatic repair processes, and other measures. The alarm data contains detailed information about the vulnerability threat, such as component name, vulnerability description, risk coefficient, etc.

[0039] The technical effect of the above technical solution is: by matching with the preset vulnerability database, the system can quickly identify open source components with known vulnerabilities, thereby improving the accuracy and efficiency of vulnerability identification. By calculating the specific risk factor, the system can provide a quantitative risk assessment result for each identified vulnerability. This can help administrators prioritize vulnerabilities according to the severity of the risk. The system can automatically mark and alert open source components with vulnerability threats, thereby reducing the need for manual intervention. This can speed up vulnerability response and reduce potential security risks. Through timely vulnerability identification and alerting, the system can promptly discover and handle potential security threats. This can enhance the overall security of the system and protect sensitive data and business from damage. The vulnerability identification and risk assessment functions of the system can be combined with actual monitoring to achieve continuous monitoring and updating. This means that the system can continuously adapt to new vulnerabilities and threats to maintain its security and effectiveness.

[0040] In one embodiment of the present invention, the S4 includes: When the update identification data is greater than or equal to the preset update identification threshold, and the threat warning data is greater than or equal to the preset warning threshold, the data collection frequency of the open source component is increased; at this time, relatively speaking, the update identification data is more likely to be due to the low model accuracy, and the threat warning data is large. At this time, the data collection frequency should be increased to optimize the model and increase the data detection density. This method mainly starts from optimizing vulnerability detection.

[0041] When the update identification data is less than the preset update identification threshold and the threat warning data is less than the preset warning threshold, the data collection frequency of the open source component is reduced; at this time, the model accuracy is high and the threat warning is small, so there is no need for an excessively high data collection frequency; this method mainly starts from optimizing vulnerability detection.

[0042] When the update identification data is greater than or equal to the preset update identification threshold and the threat alarm data is less than the preset alarm threshold, the data collection frequency of the open source component is maintained; although the model accuracy may be low, the threat alarm data is small, so the collection frequency is maintained.

[0043] When the update identification data is less than the preset update identification threshold, and the threat alarm data is greater than or equal to the preset alarm threshold, the data collection frequency of the open source component is increased or maintained. If the model accuracy is high and the threat alarm is large, you can choose to increase the data collection frequency to intensively monitor the threat data, maintain the collection frequency, and reduce the collection resource consumption when the model accuracy is high.

[0044] The working principle of the above technical solution is: when the update identification data is greater than or equal to the preset update identification threshold, and the threat warning data is greater than or equal to the preset warning threshold, the system believes that the model accuracy may be low and faces a high security threat. Therefore, the system will increase the data collection frequency of the open source component to increase the amount of data for optimizing the model and improving the density of vulnerability detection. When the update identification data is less than the preset update identification threshold, and the threat warning data is less than the preset warning threshold, the system believes that the model accuracy is high and the current security threat is low. Therefore, the system will reduce the data collection frequency of the open source component to reduce unnecessary resource consumption. When the update identification data is greater than or equal to the preset update identification threshold, but the threat warning data is less than the preset warning threshold, the system believes that although the model accuracy may be low, the current security threat is not great. Therefore, the system will maintain the current data collection frequency, neither excessively increasing resource consumption nor ignoring the need for model optimization. When the update identification data is less than the preset update identification threshold, but the threat warning data is greater than or equal to the preset warning threshold, the system faces a contradictory situation: the model accuracy is high, but the security threat is large. At this point, the system can choose to increase the frequency of data collection to monitor threat data more intensively; or choose to maintain the current frequency of data collection to minimize resource consumption while maintaining model accuracy. Specific decisions may be based on factors such as the real-time load of the system and resource availability. The entire mechanism is based on optimizing vulnerability detection and balances the relationship between model optimization and resource consumption by dynamically adjusting the frequency of data collection. When facing high threats, increase data collection to increase detection density; when facing low threats, reduce data collection to save resources.

[0045] The technical effect of the above technical solution is: by dynamically adjusting the data collection frequency, the system can quickly increase the amount of data in high-threat situations, which is used to optimize the model and improve the accuracy of vulnerability detection, and can timely discover and deal with potential security threats. In the case of low threat or high model accuracy, the system can automatically reduce the frequency of data collection, thereby reducing unnecessary resource consumption. This helps to extend the system operation time and reduce operating costs. The mechanism allows the system to dynamically adjust the data collection strategy according to the real-time situation, enhancing the flexibility and adaptability of the system. This helps the system better respond to changing security threats and business needs. By optimizing vulnerability detection and saving resource consumption, the system can provide more stable and secure services. This helps to enhance user trust and satisfaction with the system. The mechanism can be combined with the system's continuous monitoring function to achieve real-time data collection frequency adjustment and vulnerability detection optimization. This helps the system to continuously adapt to new security threats and business needs and maintain its long-term effectiveness and security. A more accurate and efficient protection method is provided.

[0046] In one embodiment of the present invention, the system comprises: The collection and identification training module is used to collect and process the data of open source components through network operation traffic data, obtain component collection and processing data, obtain feature codes and software packages to identify open source components, and train component identification models to obtain specific identification data; The model incremental training module is used to calculate the model recognition accuracy, determine whether to perform incremental training of the component recognition model, obtain the incremental recognition model through incremental training, and update the recognition data; The matching risk warning module is used to match all identification data with the preset vulnerability database for specific open source components, obtain specific matching data, calculate the specific risk coefficient of the specific open source component, and then obtain the threat warning data of the specific open source component with threat warning annotation; The dual comparison and adjustment module is used to compare the update identification data with the preset update identification threshold, and to compare the threat alarm data with the preset alarm threshold, and to adjust the data collection frequency according to the obtained threshold comparison information to obtain frequency adjustment data.

[0047] The working principle of the above technical solution is: collect relevant information through network operation traffic data, which includes the usage of software components, communication mode, etc. Process these raw data to extract the collected and processed data that can be used for identification, such as component name, version, call frequency, etc. Use feature codes and software package information to identify open source components in the traffic. Feature codes can be code snippets or identifiers unique to components, and software package information contains metadata and dependencies of components. Train component recognition models through historical recognition data, and use deep learning methods to improve recognition accuracy. This model can learn the characteristics of components, so as to more accurately identify components in traffic. The deep learning designed in this application is only a method for obtaining historical recognition rules based on historical recognition information. Calculate the model recognition accuracy and evaluate the performance of the current model. If the accuracy is lower than the preset threshold, perform incremental training. Incremental training allows the model to improve its performance through new data without training from scratch, thereby obtaining an updated recognition model and recognition data. Match the identified components with the preset vulnerability database to find out whether there are known vulnerabilities. For successfully matched components, their specific risk coefficient is calculated, which takes into account factors such as the severity of the vulnerability, the popularity of the component, and the importance of the component in the system.

[0048] Based on the risk factor, components with potential threats are marked with threat alerts. Updated identification data is compared with the preset update identification threshold to evaluate the adequacy of current data collection. Threat alert data is compared with the preset alert threshold to understand the urgency of the current threat. Based on these threshold comparison information, the data collection frequency is dynamically adjusted. Based on the comparison relationship between the identification data and alert data and the threshold, the data collection frequency is increased to obtain more detailed information, or the frequency is reduced to reduce resource consumption.

[0049] The technical effect of the above technical solution is: through the combined use of feature codes and software package information, as well as the training of deep learning models, the recognition accuracy of open source components can be significantly improved. Through matching with the preset vulnerability database and risk assessment, potential threats in the system can be monitored in real time, and vulnerabilities can be discovered and handled in a timely manner. By dynamically adjusting the data collection frequency according to the threshold comparison information, it is possible to optimize resource usage and reduce unnecessary computing and storage overhead while ensuring the monitoring effect. The entire monitoring process is highly automated, reducing the need for manual intervention. At the same time, through the training and use of deep learning models, intelligent monitoring and response are achieved. Through real-time monitoring and threat alerts, security risks in the system can be discovered and handled in a timely manner, enhancing the overall security of the system.

[0050] In one embodiment of the present invention, the acquisition and recognition training module includes: Acquire network operation flow data, and collect data of open source components on the network operation flow data according to the preset open source component type information to obtain component collection data; Preprocessing the component collected data to obtain component collected and processed data; Obtaining feature codes of preset open source components and software package information in component collection and processing data; Identify the open source component according to the feature code and the software package information to obtain component identification data; Training a deep learning model according to the component identification data to obtain a component identification model; The specific open source component is identified according to the component identification model to obtain specific identification data.

[0051] The working principle of the above technical solution is: real-time traffic data is captured from the network environment. This data usually contains a large amount of network communication information, including the content, source and destination, timestamp, etc. of the data packet. Using the preset open source component type information, the network operation traffic data is screened and analyzed to identify and extract data related to these open source components. This process involves deep analysis of the traffic data to identify specific protocols, port numbers, data formats, etc., so as to determine which traffic is related to specific open source components. The collected component data is pre-processed such as cleaning, formatting and normalization to improve the quality and availability of the data. Preprocessing may include removing redundant data, filling missing values, converting data types, etc. The signature is the key information for identifying a specific open source component, usually a set of unique code snippets or identifiers. The package information contains the metadata of the component, such as version number, dependencies, etc. This information can be obtained from the official documentation, code library or security database of the open source component. Combining the signature and package information, a deep learning method is used to identify the pre-processed component collection data, aiming to determine which parts of the traffic data include specific open source component information. Using the identified component data as training samples, a deep learning model is trained. This model can learn the feature representation of components and is used to automatically identify and classify new traffic data. The selection and training process of deep learning models involves a variety of methods and algorithms, such as convolutional neural networks (CNNs) and recurrent neural networks (RNNs). Use the trained component identification model to identify specific open source components. This process can be online identification of real-time traffic data or offline analysis of historical data. The identification results can be used in a variety of scenarios such as security monitoring, vulnerability scanning, and risk assessment.

[0052] The technical effect of the above technical solution is: by combining the signature code and software package information for identification, and using the deep learning model for training, the accuracy of open source component identification can be significantly improved. The deep learning model can learn complex feature representations and adapt to the identification needs in different scenarios. The entire identification process is highly automated, reducing the need for manual intervention. The use of deep learning models makes the identification process more intelligent, and can adaptively adjust the identification strategy to cope with different traffic data and component types. The method supports online identification of real-time traffic data, and can promptly detect and handle potential threats. At the same time, due to the good scalability of the deep learning model, it can easily adapt to new component types and signature codes, thereby maintaining the continuous update of the identification capability. By accurately identifying open source components and monitoring their usage, potential security threats can be promptly discovered and responded to. This is of great significance for protecting the integrity, availability and confidentiality of the system.

[0053] In one embodiment of the present invention, the model incremental training module includes: Obtain all specific identification data, and calculate the model identification accuracy rate based on the specific identification data; The calculation formula of the model recognition accuracy is:

[0054] Among them, ZU is the recognition accuracy of the model, TP is the number of correct recognitions in all recognition data, YL is the total number of initial feature recognition data, GL is the total number of updated recognition data, and GL may be 0; Performing incremental training determination on the component recognition model according to the recognition accuracy of the model; Acquire incremental data of component identification data, and perform incremental training on the component identification model through the incremental data to obtain an incremental identification model; Identify the incremental open source components according to the incremental identification model to obtain update identification data; In a preset time period, the model recognition accuracy is calculated based on the updated recognition data until the incremental training is determined to be unnecessary.

[0055] The working principle of the above technical solution is as follows: collect all the identification data for specific open source components from the previous steps. These data are a direct reflection of the model recognition results, including the model's recognition of each component. Calculate the model recognition accuracy. This accuracy reflects the performance of the model on the current data set. Compare the calculated model recognition accuracy with the preset accuracy threshold. If the accuracy is lower than the threshold, it means that the performance of the model needs to be improved, so incremental training is required. Incremental training is a method of using new data to optimize model performance based on an existing model. Once it is decided to perform incremental training, new component recognition data needs to be collected as incremental data. Use these incremental data to train the existing component recognition model to obtain an updated incremental recognition model. Use the incremental recognition model to identify the new incremental open source component and obtain updated recognition data. Calculate the model recognition accuracy again within a preset time period to evaluate the effect of incremental training. Repeat the above steps, i.e., collect incremental data, perform incremental training, and calculate the accuracy, until the model recognition accuracy reaches or exceeds the preset threshold, at which time it is determined that no incremental training is required.

[0056] The technical effect of the above technical solution is that through incremental training, the model can continuously adapt to new data and scenarios, thereby improving its recognition accuracy. The effectiveness and reliability of the model in practical applications can be maintained. Compared with training the entire model from scratch, incremental training only requires fine-tuning the model, so it can significantly reduce training time and computing resource consumption.

[0057] As incremental data is continuously added and trained, the model can learn more features and patterns, thereby improving its generalization ability. This means that the model can better handle unknown or unseen data. By continuously iterating training and optimizing the model, the system can adapt to changes in the network environment and new threats more quickly. It can enhance the system's adaptability and robustness and improve overall security. The incremental training process can be combined with actual monitoring to achieve real-time updating and monitoring of the model. This means that the system can detect and handle new threats in a timely manner, improving response speed and accuracy.

[0058] In one embodiment of the present invention, the matching risk warning module includes: Obtain a preset vulnerability database, and perform identification and matching of a specific open source component with the specific identification data or the updated identification data and the preset vulnerability database to obtain specific matching data; Calculate the specific risk factor of a specific open source component through specific matching data; The calculation formula of the specific risk factor is:

[0059] Among them, TD is the specific risk factor, PC is the number of successful matches between the specific identification data and the preset vulnerability database, PZ is the total number of matches between the specific identification data and the preset vulnerability database, and Q k is the preset weight value of the abnormal type of the preset vulnerability data corresponding to the kth successfully matched specific identification data. H is used to reflect the inverse relationship of the update frequency of the open source component corresponding to the specific identification data, that is, the more frequently the component is updated, the lower its risk factor is due to this factor; Comparing the specific risk factor with a preset risk threshold to obtain risk comparison information; Performing vulnerability threat determination on the specific open source component according to the risk comparison information to obtain vulnerability threat determination information; Performing threat warning marking on the specific open source component according to the vulnerability threat determination information; Threat alerts are issued for vulnerabilities in specific open source components with threat alert annotations to obtain threat alert data.

[0060] The working principle of the above technical solution is as follows: obtain a preset vulnerability database, which contains vulnerability information of various known open source components, including vulnerability description, impact scope, repair suggestions, etc. This database is the basis for subsequent matching and risk assessment. Match specific identification data (or update identification data) with the preset vulnerability database. This process aims to identify which open source components have known vulnerabilities. Matching is based on information such as component name, version, and feature code. For the open source components identified with vulnerabilities, calculate their risk coefficients. Compare the calculated specific risk coefficients with the preset risk threshold. If the risk coefficient exceeds the threshold, it is considered that the open source component has a high vulnerability threat and needs further processing. For the open source components determined to have vulnerability threats, the system performs threat alarm annotations. This usually means marking these components in the system's management interface or log so that administrators can quickly identify and handle them. Threat alarms are issued for specific open source components with threat alarm annotations. This may include sending alarm notifications, generating alarm reports, triggering automatic repair processes, and other measures. The alarm data contains detailed information about the vulnerability threat, such as component name, vulnerability description, risk coefficient, etc.

[0061] The technical effect of the above technical solution is: by matching with the preset vulnerability database, the system can quickly identify open source components with known vulnerabilities, thereby improving the accuracy and efficiency of vulnerability identification. By calculating the specific risk factor, the system can provide a quantitative risk assessment result for each identified vulnerability. This can help administrators prioritize vulnerabilities according to the severity of the risk. The system can automatically mark and alert open source components with vulnerability threats, thereby reducing the need for manual intervention. This can speed up vulnerability response and reduce potential security risks. Through timely vulnerability identification and alerting, the system can promptly discover and handle potential security threats. This can enhance the overall security of the system and protect sensitive data and business from damage. The vulnerability identification and risk assessment functions of the system can be combined with actual monitoring to achieve continuous monitoring and updating. This means that the system can continuously adapt to new vulnerabilities and threats to maintain its security and effectiveness.

[0062] In one embodiment of the present invention, the dual comparison adjustment module includes: When the update identification data is greater than or equal to the preset update identification threshold, and the threat warning data is greater than or equal to the preset warning threshold, the data collection frequency of the open source component is increased; at this time, relatively speaking, the update identification data is more likely to be due to the low model accuracy, and the threat warning data is large. At this time, the data collection frequency should be increased to optimize the model and increase the data detection density. This method mainly starts from optimizing vulnerability detection.

[0063] When the update identification data is less than the preset update identification threshold and the threat warning data is less than the preset warning threshold, the data collection frequency of the open source component is reduced; at this time, the model accuracy is high and the threat warning is small, so there is no need for an excessively high data collection frequency; this method mainly starts from optimizing vulnerability detection.

[0064] When the update identification data is greater than or equal to the preset update identification threshold and the threat alarm data is less than the preset alarm threshold, the data collection frequency of the open source component is maintained; although the model accuracy may be low, the threat alarm data is small, so the collection frequency is maintained.

[0065] When the update identification data is less than the preset update identification threshold, and the threat alarm data is greater than or equal to the preset alarm threshold, the data collection frequency of the open source component is increased or maintained. If the model accuracy is high and the threat alarm is large, you can choose to increase the data collection frequency to intensively monitor the threat data, maintain the collection frequency, and reduce the collection resource consumption when the model accuracy is high.

[0066] The working principle of the above technical solution is: when the update identification data is greater than or equal to the preset update identification threshold, and the threat warning data is greater than or equal to the preset warning threshold, the system believes that the model accuracy may be low and faces a high security threat. Therefore, the system will increase the data collection frequency of the open source component to increase the amount of data for optimizing the model and improving the density of vulnerability detection. When the update identification data is less than the preset update identification threshold, and the threat warning data is less than the preset warning threshold, the system believes that the model accuracy is high and the current security threat is low. Therefore, the system will reduce the data collection frequency of the open source component to reduce unnecessary resource consumption. When the update identification data is greater than or equal to the preset update identification threshold, but the threat warning data is less than the preset warning threshold, the system believes that although the model accuracy may be low, the current security threat is not great. Therefore, the system will maintain the current data collection frequency, neither excessively increasing resource consumption nor ignoring the need for model optimization. When the update identification data is less than the preset update identification threshold, but the threat warning data is greater than or equal to the preset warning threshold, the system faces a contradictory situation: the model accuracy is high, but the security threat is large. At this point, the system can choose to increase the frequency of data collection to monitor threat data more intensively; or choose to maintain the current frequency of data collection to minimize resource consumption while maintaining model accuracy. Specific decisions may be based on factors such as the real-time load of the system and resource availability. The entire mechanism is based on optimizing vulnerability detection and balances the relationship between model optimization and resource consumption by dynamically adjusting the frequency of data collection. When facing high threats, increase data collection to increase detection density; when facing low threats, reduce data collection to save resources.

[0067] The technical effect of the above technical solution is: by dynamically adjusting the data collection frequency, the system can quickly increase the amount of data in high-threat situations, which is used to optimize the model and improve the accuracy of vulnerability detection, and can timely discover and deal with potential security threats. In the case of low threat or high model accuracy, the system can automatically reduce the frequency of data collection, thereby reducing unnecessary resource consumption. This helps to extend the system operation time and reduce operating costs. The mechanism allows the system to dynamically adjust the data collection strategy according to the real-time situation, enhancing the flexibility and adaptability of the system. This helps the system better respond to changing security threats and business needs. By optimizing vulnerability detection and saving resource consumption, the system can provide more stable and secure services. This helps to enhance user trust and satisfaction with the system. The mechanism can be combined with the system's continuous monitoring function to achieve real-time data collection frequency adjustment and vulnerability detection optimization. This helps the system to continuously adapt to new security threats and business needs and maintain its long-term effectiveness and security. A more accurate and efficient protection method is provided.

[0068] Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A method for monitoring open source component vulnerability traffic, characterized in that: The method comprises: S1. Collect and process data of open source components through network operation traffic data, obtain component collection and processing data, obtain feature codes and software packages to identify open source components, and train component identification models to obtain specific identification data; S2. Calculate the model recognition accuracy, determine whether to perform incremental training of the component recognition model, obtain the incremental recognition model and update the recognition data through incremental training; S3, matching all identification data with a preset vulnerability database for specific open source components, obtaining specific matching data, calculating a specific risk factor for the specific open source component, and then obtaining threat warning data for the specific open source component with a threat warning label; S4. Compare the update identification data with a preset update identification threshold, and compare the threat warning data with a preset warning threshold, and adjust the data collection frequency according to the obtained threshold comparison information to obtain frequency adjustment data.

2. According to claim 1, a method for monitoring open source component vulnerability traffic is characterized in that: The S1 includes: Acquire network operation flow data, and collect data of open source components on the network operation flow data according to the preset open source component type information to obtain component collection data; Preprocessing the component collected data to obtain component collected and processed data; Obtaining feature codes of preset open source components and software package information in component collection and processing data; Identify the open source component according to the feature code and the software package information to obtain component identification data; Training a deep learning model according to the component identification data to obtain a component identification model; The specific open source component is identified according to the component identification model to obtain specific identification data.

3. According to claim 1, a method for monitoring open source component vulnerability traffic is characterized in that: The S2 includes: Obtain all specific identification data, and calculate the model identification accuracy rate based on the specific identification data; Performing incremental training determination on the component recognition model according to the recognition accuracy of the model; Acquire incremental data of component identification data, and perform incremental training on the component identification model through the incremental data to obtain an incremental identification model; Identify the incremental open source components according to the incremental identification model to obtain update identification data; In a preset time period, the model recognition accuracy is calculated based on the updated recognition data until the incremental training is determined to be unnecessary.

4. According to claim 1, a method for monitoring open source component vulnerability traffic is characterized in that: The S3 includes: Obtain a preset vulnerability database, and perform identification and matching of a specific open source component with the specific identification data or the updated identification data and the preset vulnerability database to obtain specific matching data; Calculate the specific risk factor of a specific open source component through specific matching data; Comparing the specific risk factor with a preset risk threshold to obtain risk comparison information; Performing vulnerability threat determination on the specific open source component according to the risk comparison information to obtain vulnerability threat determination information; Performing threat warning marking on the specific open source component according to the vulnerability threat determination information; Threat alerts are issued for vulnerabilities in specific open source components with threat alert annotations to obtain threat alert data.

5. According to claim 1, a method for monitoring open source component vulnerability traffic is characterized in that: The S4 includes: When the update identification data is greater than or equal to the preset update identification threshold, and the threat warning data is greater than or equal to the preset warning threshold, the data collection frequency of the open source component is increased; When the update identification data is less than the preset update identification threshold and the threat warning data is less than the preset warning threshold, the data collection frequency of the open source component is reduced; When the update identification data is greater than or equal to the preset update identification threshold, and the threat warning data is less than the preset warning threshold, the data collection frequency of the open source component is maintained; When the update identification data is less than a preset update identification threshold and the threat warning data is greater than or equal to the preset warning threshold, the data collection frequency of the open source component is increased or maintained.

6. An open source component vulnerability traffic monitoring system, characterized in that: The system comprises: The collection and identification training module is used to collect and process the data of open source components through network operation traffic data, obtain component collection and processing data, obtain feature codes and software packages to identify open source components, and train component identification models to obtain specific identification data; The model incremental training module is used to calculate the model recognition accuracy, determine whether to perform incremental training of the component recognition model, obtain the incremental recognition model through incremental training, and update the recognition data; The matching risk warning module is used to match all identification data with the preset vulnerability database for specific open source components, obtain specific matching data, calculate the specific risk coefficient of the specific open source component, and then obtain the threat warning data of the specific open source component with threat warning annotation; The dual comparison and adjustment module is used to compare the update identification data with the preset update identification threshold, and to compare the threat alarm data with the preset alarm threshold, and to adjust the data collection frequency according to the obtained threshold comparison information to obtain frequency adjustment data.

7. The open source component vulnerability flow monitoring system according to claim 6, characterized in that: The acquisition and recognition training module includes: Acquire network operation flow data, and collect data of open source components on the network operation flow data according to the preset open source component type information to obtain component collection data; Preprocessing the component collected data to obtain component collected and processed data; Obtaining feature codes of preset open source components and software package information in component collection and processing data; Identify the open source component according to the feature code and the software package information to obtain component identification data; Training a deep learning model according to the component identification data to obtain a component identification model; The specific open source component is identified according to the component identification model to obtain specific identification data.

8. The open source component vulnerability flow monitoring system according to claim 6, characterized in that: The model incremental training module includes: Obtain all specific identification data, and calculate the model identification accuracy rate based on the specific identification data; Performing incremental training determination on the component recognition model according to the recognition accuracy of the model; Acquire incremental data of component identification data, and perform incremental training on the component identification model through the incremental data to obtain an incremental identification model; Identify the incremental open source components according to the incremental identification model to obtain update identification data; In a preset time period, the model recognition accuracy is calculated based on the updated recognition data until the incremental training is determined to be unnecessary.

9. The open source component vulnerability flow monitoring system according to claim 6, characterized in that: The matching risk warning module includes: Obtain a preset vulnerability database, and perform identification and matching of a specific open source component with the specific identification data or the updated identification data and the preset vulnerability database to obtain specific matching data; Calculate the specific risk factor of a specific open source component through specific matching data; Comparing the specific risk factor with a preset risk threshold to obtain risk comparison information; Performing vulnerability threat determination on the specific open source component according to the risk comparison information to obtain vulnerability threat determination information; Performing threat warning marking on the specific open source component according to the vulnerability threat determination information; Threat alerts are issued for vulnerabilities in specific open source components with threat alert annotations to obtain threat alert data.

10. The open source component vulnerability flow monitoring system according to claim 6, characterized in that: The dual comparison adjustment module comprises: When the update identification data is greater than or equal to the preset update identification threshold, and the threat warning data is greater than or equal to the preset warning threshold, the data collection frequency of the open source component is increased; When the update identification data is less than the preset update identification threshold and the threat warning data is less than the preset warning threshold, the data collection frequency of the open source component is reduced; When the update identification data is greater than or equal to the preset update identification threshold, and the threat warning data is less than the preset warning threshold, the data collection frequency of the open source component is maintained; When the update identification data is less than a preset update identification threshold and the threat warning data is greater than or equal to the preset warning threshold, the data collection frequency of the open source component is increased or maintained.

Images