Internet unrecorded system monitoring device and method based on deep learning and multi-network space search engine, terminal equipment and storage medium

Through the combination of deep learning and multi-cyberspace search engines, efficient and comprehensive monitoring of unregistered systems in the Internet is achieved, and the problems of low retrieval efficiency and limited scope in the existing technology are solved, which significantly improves network security.

CN119089040APending Publication Date: 2024-12-06INFORMATION & COMM CO OF STATE GRID XINJIANG ELECTRIC POWER CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411136525.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-19
Publication Date
2024-12-06

AI Technical Summary

Technical Problem

The existing technology is inefficient and limited in scope when retrieving unregistered systems on the Internet, making it difficult for enterprises to detect and deal with potential threats in a timely manner, increasing network security risks.

Method used

Adopt a monitoring device based on deep learning and multi-cyberspace search engines, and call multiple search engine APIs through the data acquisition module. The intelligent filtering module uses the deep learning model to identify relevant information, and eliminates non-related data through the whitelist module to achieve automation and efficiency. system monitoring.

Benefits of technology

It significantly improves the search efficiency and accuracy, expands the monitoring range, reduces repetitive work, improves the identification ability of unregistered systems, and reduces network security risks.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119089040A_ABST
    Figure CN119089040A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of network security, in particular to an internet unrecorded system monitoring device and method based on deep learning and a multi-network space search engine, terminal equipment and a storage medium, which extract text information from a system page, encode the text information into a vector after data preprocessing, and send the vector to a server; the method comprises the following steps of: encoding a character vector, processing the encoded character vector, identifying characters related to enterprise businesses, predicting system characteristic data in a retrieval result, and judging whether a system is related to an enterprise or not, thereby intelligently screening webpage contents and automatically identifying information related to the enterprise. According to the method, the webpage content is intelligently screened, information related to enterprises is automatically identified, a large number of business systems irrelevant to the enterprises are removed, and the retrieval accuracy and efficiency are improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of network security technology, and is an Internet unregistered system monitoring device, method, terminal equipment and storage medium based on deep learning and multi-network space search engine. Background Art

[0002] With the rapid development of information technology, many companies have launched various online service platforms to improve customer service experience. However, while this trend enriches the service content, it also makes companies face rising network security risks. In order to ensure the stability and security of business processes, many companies have begun to implement strict information system management processes to ensure that the system meets high standards of security requirements in all aspects such as design, deployment and operation. However, in actual operations, some third-party developers and corporate employees may choose to violate relevant regulations and privately deploy systems that have not been strictly reviewed on external networks in pursuit of efficiency or because the relevant systems cannot meet the requirements of the company, thereby evading the formal approval process. However, these systems that have not been filed by the company usually have gaps in network security protection and are easily targeted by hackers. Once an attack occurs, it may lead to the leakage of sensitive data, causing immeasurable economic losses and reputation damage to the company.

[0003] In response to this problem, many companies currently use cyberspace search engines such as FOFA and Hunter to manually input keywords closely related to the business to check whether there are systems that have been privately launched without filing on the Internet. Although this method is intuitive and easy to use, there are still several problems. First, it is time-consuming and inefficient. During the search process, the search is usually roughly based on specific keywords in the system title or page. Therefore, the results returned by the search engine are mixed with a large number of other business systems and advertising websites that are not related to the company's business, resulting in a lot of time required for screening and identification. Second, the scope of this method is limited. A single cyberspace search engine is often difficult to cover all business systems on the Internet; in order to conduct a comprehensive investigation, companies usually need to repeat the same search operation in multiple cyberspace search engines; but as the number of search engines used increases, the investigation time will also increase exponentially, which seriously limits the company's ability to further expand the scope of investigation.

[0004] In summary, although manual search and screening of unregistered online systems on the Internet is intuitive and easy to implement, enterprises still face certain challenges in actual operation due to its time-consuming, low efficiency and limited screening scope. Therefore, it is necessary to explore more efficient and comprehensive methods to timely discover and deal with unregistered online systems on the Internet to solve the above problems. Summary of the invention

[0005] The present invention provides an Internet unregistered system monitoring device, method, terminal equipment and storage medium based on deep learning and multi-network space search engines, which overcomes the shortcomings of the above-mentioned prior art and can effectively solve the problems of long time, low efficiency and limited investigation scope faced by traditional manual retrieval and investigation of unregistered systems on the Internet.

[0006] One of the technical solutions of the present invention is achieved by the following measures: an Internet unregistered system monitoring device based on deep learning and multi-network space search engine, including a data acquisition module, an intelligent screening module and a whitelist module;

[0007] The data collection module is used to call multiple search engine APIs to retrieve data, obtain the whitelist from the whitelist module, and remove the systems in the whitelist from the search results. It accesses the system page by calling the browser kernel, collects relevant information in the page, and completes data collection.

[0008] Among them, the intelligent screening module is used to extract text information from the system page, encode the text information into a vector after data preprocessing, process the encoded text vector, identify the text related to the enterprise business, predict the system feature data in the search results, and determine whether the system and the enterprise are related, so as to intelligently screen the web page content and automatically identify the information related to the enterprise;

[0009] Among them, the whitelist module is used to store the registered system information and record the information irrelevant to the enterprise that was eliminated by the last deep learning-based intelligent screening module.

[0010] The following is a further optimization and / or improvement of one of the above-mentioned technical solutions:

[0011] The above-mentioned data collection module may include a data retrieval component, a whitelist data processing component and a system page information collection component; the data retrieval component is used to retrieve data by calling multiple different search engine APIs; the whitelist data processing component is used to obtain a whitelist list from the whitelist module and eliminate the systems in the whitelist from the retrieval results; the system page information collection component is used to access the system page by calling the browser kernel and collect relevant information in the page.

[0012] The above-mentioned intelligent screening module may include a data preprocessing component and a deep learning component; wherein, the data preprocessing component is used to extract text information from the system page, and after simple preprocessing, encode the text information into a vector; wherein, the deep learning component is used to process the encoded text vector, identify text related to the enterprise's business, predict the system feature data in the search results, and determine whether the system and the enterprise are related.

[0013] The second technical solution of the present invention is achieved by the following measures: a method for monitoring an Internet unregistered system based on deep learning and multi-network space search engines, comprising:

[0014] Step S1: Call multiple different search engine APIs to retrieve data, obtain a whitelist from the whitelist module, and remove the systems in the whitelist from the search results, access the system page by calling the browser kernel, collect relevant information in the page, and complete data collection;

[0015] Step S2: extract text information from the system page, encode the text information into a vector after data preprocessing, process the encoded text vector, identify the text related to the enterprise business, predict the system feature data in the search results, and determine whether the system and the enterprise are related, so as to intelligently screen the web page content and automatically identify information related to the enterprise.

[0016] The following is a further optimization and / or improvement of the second technical solution of the above invention:

[0017] In the above step S1, data collection may specifically include the following steps:

[0018] Step S1.1: read all keywords from the keyword list, and simultaneously start the automated retrieval program of different network space search engines to retrieve all keywords; after the retrieval is completed, aggregate all the retrieval results into an intermediate file;

[0019] Step S1.2: Taking the system IP, port and system title as input, calculate the MD5 value of each system;

[0020] Step S1.3: Obtain the MD5 values ​​of all systems in the whitelist, compare them with the MD5 values ​​in the search results, and remove systems with the same MD5 values;

[0021] Step S1.4: Start the browsing kernel, browse all remaining system pages, collect information from the pages, and then write all collected data into a file.

[0022] In the above step S2, when performing data preprocessing, the following steps may be specifically included:

[0023] Step S2.1: extract text information from the system page, and remove invalid words after screening; combine the screened text information with the system title to form the features of the system, and set the extracted features as D;

[0024] Step S2.2: Use One-Hot encoding to encode the system features and map the characters in the text feature D. The vector representation of the mth word representing the features of system i.

[0025] In the above step S2, when predicting the system characteristic data in the search results, the following steps may be specifically included:

[0026] Step S2.1: Transform the characteristic matrix X of system i into i The input is fed into the deep bidirectional language model pre-trained by Bert, which embeds the meaning of words and the relationship between words into the vector representation; the computational complexity of Bert is O(n 2 ), n is the length of the input feature; X i Divide into J segments according to a certain length, and input each segment into the Bert model separately;

[0027]

[0028] Step S2.2: Input to the linear binary classifier and predict Is each word related to the business of the enterprise? If it is relevant, it is predicted as 1, and if it is not relevant, it is predicted as 0. After the prediction is completed, the prediction result is To splice;

[0029]

[0030]

[0031] In the above formula, and b1 represent learnable parameters, Represents a splicing operation;

[0032] Step S2.3: Splice and input into the Bi-directional Long Short-Term Memory (BILSTM) model to learn the vector representation of all text features in the system X *i ;

[0033]

[0034] Step S2.4: The vector representation X of the entire text feature of the system *i Input to the linear binary classifier to predict whether the system is relevant to the enterprise business. If relevant, the prediction is 1, and if not, the prediction is 0;

[0035]

[0036] In the above formula, and b2 represent learnable parameters respectively;

[0037] Step S2.5: Based on the prediction results in step S2.2, calculate the proportion of words related to the enterprise business in the entire text features of the system, and set the proportion to λ i ;

[0038] Step S2.6: The value of is 1 and λ i The system that is greater than the threshold λ set by the algorithm is predicted to be an unregistered corporate website, and the words predicted by the deep learning model in the system text features are marked according to step S2.2.

[0039] In the above step S2, before using the deep learning model to automatically screen the data, it is necessary to optimize the parameters using the data that has been manually annotated in advance, which may specifically include the following steps:

[0040] Step S2.1.1: Select l systems that have been registered by the enterprise and l other systems that are not related to the enterprise, perform data preprocessing, extract information from the registered systems to construct positive sample data X + , extract information from other irrelevant systems to construct negative sample data X - ;

[0041] Step S2.1.2: Set X + and X - Mix them together to construct the model input data X, and mark the words in X according to whether they are related to the business of the enterprise, and construct the word-level label vector Y w , and at the same time, the elements in X belong to X + The system is marked as 1, belonging to X - The system feature is marked as 0, and the entire text level label vector Y of the system is constructed p ;

[0042] Step S2.1.3: Divide X and Y into training data sets in a ratio of 8:2 Test Dataset

[0043] Step S2.1.4: Use the deep learning model f to learn X train The feature information is used to predict the word level and obtain the prediction result Y wi and system-level prediction result Y p,i ;

[0044]

[0045] Step S2.1.5: Use the binary cross entropy loss function to calculate the prediction results separately and the true data label Loss value between w and loss p ;

[0046]

[0047]

[0048] Step S2.1.6: Take the loss value loss w and loss p As input, the gradient of the model parameters is calculated and the parameters of the model f are optimized using the multi-objective MTAdam optimizer

[0049]

[0050] f' θ =MTAdam(loss w , loss p , f θ )

[0051] Step S2.1.7: Predict X using the optimized model f test , and then the prediction results and the true data label Compare and calculate the prediction accuracy of the model. If the accuracy is greater than the current optimal result, save the model parameters.

[0052] Step S2.1.8: Repeat the above steps, iterate k times, gradually optimize the model parameters, and obtain the parameter values ​​under the optimal state of the model.

[0053] The third technical solution of the present invention is achieved through the following measures: a terminal device includes a memory and a processor, the memory stores a program that can be run on the processor, and when the processor executes the program, it implements the above-mentioned Internet unregistered system monitoring method based on deep learning and multi-network space search engines.

[0054] The fourth technical solution of the present invention is achieved through the following measures: a storage medium, the storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the above-mentioned Internet unregistered system monitoring method based on deep learning and multi-network space search engines.

[0055] The present invention integrates multiple cyberspace search engines by developing an API-based automated retrieval program, and uses a deep learning model to intelligently screen the retrieval results, effectively excluding irrelevant data, thereby constructing an efficient and comprehensive unregistered system monitoring device. The present invention designs an automated retrieval program based on the search engine API to achieve simultaneous retrieval of multiple search engines, while improving retrieval efficiency and significantly expanding the scope of retrieval. The present invention uses a deep learning model to intelligently screen web page content, automatically identify information related to the enterprise, and eliminate a large number of business systems that are not related to the enterprise, thereby improving the accuracy and efficiency of retrieval. In addition, through the whitelist module, system information that has been confirmed to be irrelevant to the enterprise or has been registered is included in the whitelist, and these systems are automatically excluded in subsequent retrievals, reducing repetitive work and further improving overall monitoring efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 It is a schematic diagram of the overall structure of an Internet unregistered system monitoring device based on deep learning and multi-network space search engines in an embodiment of the present invention.

[0057] Figure 2 Schematic diagram of the structure of the data acquisition module in the embodiment of the present invention.

[0058] Figure 3 Schematic diagram of the structure of the intelligent screening module in an embodiment of the present invention. DETAILED DESCRIPTION

[0059] The present invention is not limited by the following embodiments, and specific implementation methods can be determined based on the technical solution of the present invention and actual conditions.

[0060] The present invention will be further described below in conjunction with embodiments:

[0061] Embodiment 1: As attached Figure 1 , 2 As shown in 3, the Internet unregistered system monitoring device based on deep learning and multi-network space search engine includes a data collection module, an intelligent screening module and a whitelist module;

[0062] Among them, the data acquisition module is used to call multiple different search engine APIs to retrieve data, obtain the whitelist list from the whitelist module, and remove the systems in the whitelist from the search results, access the system page by calling the browser kernel, collect relevant information in the page, and complete data collection. Among them, the data acquisition module includes a data retrieval component, a whitelist data processing component and a system page information collection component; in the present invention, the data retrieval component is used to retrieve data by calling multiple different search engine APIs; the whitelist data processing component is used to obtain the whitelist list from the whitelist module and remove the systems in the whitelist from the search results; the system page information collection component is used to access the system page by calling the browser kernel and collect relevant information in the page. Among them, in the present invention, the specific process of data collection is: 1) read all keywords from the keyword list, and simultaneously start the automated retrieval program of different network space search engines to retrieve all keywords. After the retrieval is completed, all retrieval results are summarized into an intermediate file. 2) Take the system IP, port and system title as input, and calculate the MD5 value of each system. 3) Get the MD5 values ​​of all systems in the whitelist list, and compare them with the MD5 values ​​in the search results, and remove the systems with the same MD5 values. 4) Use Playwright to start the browsing kernel, browse all remaining system pages, collect information from the pages, and then write all collected data into a file.

[0063] The intelligent screening module is used to extract text information from the system page, encode the text information into a vector after data preprocessing, process the encoded text vector, identify the text related to the enterprise business, predict the system feature data in the search results, and judge whether the system and the enterprise are related, so as to intelligently screen the web page content and automatically identify the information related to the enterprise. The intelligent screening module includes a data preprocessing component and a deep learning component; the data preprocessing component is used to extract text information from the system page, and encode the text information into a vector after simple preprocessing; the deep learning component is used to process the encoded text vector, identify the text related to the enterprise business, predict the system feature data in the search results, and judge whether the system and the enterprise are related.

[0064] In the present invention, the specific process of data preprocessing is as follows: 1) extract text information from the system page, and remove invalid words such as "login" and "register" after screening. Then, the screened text information is spliced ​​with the system title to form the system feature, and the extracted feature is set as D. 2) Use One-Hot encoding to encode the system feature, and map the characters in the text feature D. The vector representation of the mth word representing the features of system i.

[0065] In the present invention, the deep learning component uses the model f designed by the present invention to predict the system feature data in the search results to determine whether the system is related to the enterprise. The specific prediction process is:

[0066] 1) The characteristic matrix X of system i i The input is fed into the pre-trained deep bidirectional language model of Bert (Bidirectional Encoder Representations from Transformers), which embeds the meaning of the words in the text and the relationship between the words into the vector representation. The computational complexity of Bert is O(n 2 ), n is the length of the input feature. In order to improve the calculation speed, X i It is divided into J segments according to a certain length, and each segment is input into the Bert model separately.

[0067]

[0068] 2) Input to the linear binary classifier and predict Each word in the prediction is related to the business of the enterprise. If it is relevant, it is predicted as 1, and if it is not relevant, it is predicted as 0. After the prediction is completed, the prediction result is Make stitching.

[0069]

[0070]

[0071] In the above formula, and b1 represent learnable parameters, Represents a concatenation operation.

[0072] 3) Splice and input into the Bilstm (Bi-directional Long Short-Term Memory) model to learn the vector representation X of all text features of the system *i .

[0073]

[0074] 4) Similarly, the vector representation of the entire text feature of the system is X *i The input is sent to a linear binary classifier to predict whether the system is relevant to the enterprise business. If it is relevant, the prediction is 1, and if it is not relevant, the prediction is 0.

[0075]

[0076] In the above formula, and b2 represent learnable parameters respectively.

[0077] 5) Based on the prediction results in 2), calculate the proportion of words related to the business of the entire text features of the system, and set the proportion to λ i .

[0078] 6) The value of is 1 and λ i Systems with values ​​greater than the threshold λ set by the algorithm are predicted as unregistered corporate websites, and the words predicted by the deep learning model in the system text features are marked in red according to 2). Other systems that do not meet the conditions are added to the whitelist list of the whitelist module.

[0079] From the above steps, it can be found that the model designed by the present invention involves the learnable parameters Bert of the Bert model. θ , the learnable parameters of the Bilstm model θ and the learnable parameters of the two linear classifiers b1. b2. Therefore, before using the deep learning model to automatically filter data, it is necessary to optimize these parameters using data that has been manually labeled in advance. The specific optimization process is:

[0080] 1) Select l systems that have been registered by the enterprise and l other systems that are not related to the enterprise, and use the data preprocessing method designed by the present invention to extract information from the registered systems to construct positive sample data X + , extract information from other irrelevant systems to construct negative sample data X - .

[0081] 2) X + and X - Mix them together to construct the model input data X, and mark the words in X according to whether they are related to the business of the enterprise, and construct the word-level label vector Y w , and at the same time, the elements in X belong to X + The system is marked as 1, belonging to X - The system feature is marked as 0, and the entire text level label vector Y of the system is constructed p .

[0082] 3) Divide X and Y into training data sets in a ratio of 8:2 Test Dataset

[0083] 4) Use the deep learning model f to calculate X train The feature information is used to predict the word level and obtain the prediction result Y wi and system-level prediction result Y p,i .

[0084]

[0085] 5) Use the binary cross entropy loss function to calculate the prediction results separately and the true data label Loss value between w and loss p .

[0086]

[0087]

[0088] 6) Loss value loss w and loss p As input, the gradient of the model parameters is calculated and the parameters of the model f are optimized using the multi-objective MTAdam optimizer

[0089] f' θ =MTAdam(loss w , loss p , f θ )

[0090] 7) Use the optimized model f to predict X test , and then the prediction results and the true data label Compare and calculate the model prediction accuracy. If the accuracy is greater than the current optimal result, save the model parameters.

[0091] 8) Repeat the above steps, iterate k times, gradually optimize the model parameters, and obtain the parameter values ​​of the model in the optimal state. k can be between [10,50] according to the difficulty of data training.

[0092] The whitelist module is used to store the registered system information and record the information that is irrelevant to the enterprise that was removed by the intelligent screening module based on deep learning last time. The whitelist module plays a key role as a memory unit in the system, and includes the system information that has been confirmed to be irrelevant to the enterprise or has been registered in the whitelist. In the subsequent data collection process, the whitelist module will identify and delete these invalid data, reduce repetitive work, and improve the overall operation efficiency of the system.

[0093] This embodiment integrates multiple cyberspace search engines by developing an API-based automated search program, and uses a deep learning model to intelligently screen the search results, effectively excluding irrelevant data, thereby building an efficient and comprehensive unregistered system monitoring device. An automated search program based on the search engine API is designed to achieve simultaneous retrieval of multiple search engines, which improves retrieval efficiency and significantly expands the scope of retrieval. Intelligent screening of web page content, automatic identification of information related to the enterprise, and elimination of a large number of business systems unrelated to the enterprise, improving the accuracy and efficiency of retrieval. In addition, through the whitelist module, system information that has been confirmed to be unrelated to the enterprise or has been registered is included in the whitelist, and these systems are automatically excluded in subsequent searches, reducing repetitive work and further improving overall monitoring efficiency.

[0094] Embodiment 2: This embodiment provides an Internet unregistered system monitoring method based on deep learning and multi-network space search engines, including:

[0095] Step S1: Call multiple different search engine APIs to retrieve data, obtain the whitelist list from the whitelist module, and remove the systems in the whitelist from the search results, access the system page by calling the browser kernel, collect relevant information in the page, and complete data collection. In step S1, data collection specifically includes the following steps:

[0096] Step S1.1: read all keywords from the keyword list, and simultaneously start the automated retrieval program of different network space search engines to retrieve all keywords; after the retrieval is completed, aggregate all the retrieval results into an intermediate file;

[0097] Step S1.2: Taking the system IP, port and system title as input, calculate the MD5 value of each system;

[0098] Step S1.3: Obtain the MD5 values ​​of all systems in the whitelist, compare them with the MD5 values ​​in the search results, and remove systems with the same MD5 values;

[0099] Step S1.4: Use Playwright to start the browsing kernel, browse all remaining system pages, collect information from the pages, and then write all collected data into a file.

[0100] Step S2: extract text information from the system page, encode the text information into a vector after data preprocessing, process the encoded text vector, identify the text related to the enterprise business, predict the system feature data in the search results, and determine whether the system and the enterprise are related, so as to intelligently screen the web page content and automatically identify information related to the enterprise.

[0101] In step S2, when data preprocessing is performed, the following steps are specifically included:

[0102] Step S2.1: extract text information from the system page, and remove invalid words such as "login" and "register" after screening; then, concatenate the screened text information with the system title to form the features of the system, and set the extracted features as D;

[0103] Step S2.2: Use One-Hot encoding to encode the system features and map the characters in the text feature D. The vector representation of the mth word representing the features of system i.

[0104] In step S2, when predicting the system characteristic data in the search results, the following steps are specifically included:

[0105] Step S2.1: Transform the characteristic matrix X of system i into i The input is fed into the pre-trained deep bidirectional language model of Bert (Bidirectional Encoder Representations from Transformers), which embeds the meaning of words and the relationship between words into the vector representation; the computational complexity of Bert is O(n 2 ), n is the length of the input feature. In order to improve the calculation speed, X i Divide into J segments according to a certain length, and input each segment into the Bert model separately;

[0106]

[0107] Step S2.2: Input to the linear binary classifier and predict Each word in the prediction is related to the business of the enterprise. If it is relevant, it is predicted as 1, and if it is not relevant, it is predicted as 0. After the prediction is completed, the prediction result is To splice;

[0108]

[0109]

[0110] In the above formula, and b1 represent learnable parameters, Represents a splicing operation;

[0111] Step S2.3: Splice and input into the Bi-directional Long Short-Term Memory (BILSTM) model to learn the vector representation of all text features in the system X *i ;

[0112]

[0113] Step S2.4: Similarly, the vector representation X of the entire text feature of the system *i Input to the linear binary classifier to predict whether the system is relevant to the enterprise business. If relevant, the prediction is 1, and if not, the prediction is 0;

[0114]

[0115] In the above formula, and b2 represent learnable parameters respectively;

[0116] Step S2.5: Based on the prediction results in step S2.2, calculate the proportion of words related to the business of the entire text features of the system, and set the proportion to λ i ;

[0117] Step S2.6: The value of is 1 and λ i Systems with values ​​greater than the threshold λ set by the algorithm are predicted to be unregistered corporate websites, and the words predicted by the deep learning model in the system text features are marked according to step S2.2, for example, they can be marked in red; other systems that do not meet the conditions are added to the whitelist list of the whitelist module.

[0118] From the above steps, it can be found that the model designed by the present invention involves the learnable parameters Bert of the Bert model. θ , the learnable parameters of the Bilstm model θ and the learnable parameters of the two linear classifiers b1. Therefore, in step S2, before using the deep learning model to automatically screen the data, it is necessary to use the data that has been manually annotated in advance to optimize the parameters, which specifically includes the following steps:

[0119] Step S2.1.1: Select l systems that have been registered by the enterprise and l other systems that are not related to the enterprise, perform data preprocessing, extract information from the registered systems to construct positive sample data X + , extract information from other irrelevant systems to construct negative sample data X - ;

[0120] Step S2.1.2: Set X + and X -Mix them together to construct the model input data X, and mark the words in X according to whether they are related to the business of the enterprise, and construct the word-level label vector Y w , and at the same time, the elements in X belong to X + The system is marked as 1, belonging to X - The system feature is marked as 0, and the entire text level label vector Y of the system is constructed p ;

[0121] Step S2.1.3: Divide X and Y into training data sets in a ratio of 8:2 Test Dataset

[0122] Step S2.1.4: Use the deep learning model f to learn X train The feature information is used to predict the word level and obtain the prediction result Y wi and system-level prediction result Y p,i ;

[0123]

[0124] Step S2.1.5: Use the binary cross entropy loss function to calculate the prediction results separately and the true data label Loss value between w and loss p ;

[0125]

[0126]

[0127] Step S2.1.6: Take the loss value loss w and loss p As input, the gradient of the model parameters is calculated and the parameters of the model f are optimized using the multi-objective MTAdam optimizer

[0128]

[0129] f' θ =MTAdam(loss w , loss p , f θ )

[0130] Step S2.1.7: Predict X using the optimized model f test , and then the prediction results and the true data label Y t p estCompare and calculate the prediction accuracy of the model. If the accuracy is greater than the current optimal result, save the model parameters.

[0131] Step S2.1.8: Repeat the above steps, iterate k times, gradually optimize the model parameters, and obtain the parameter values ​​of the model in the optimal state. k can be between [10,50] according to the difficulty of data training.

[0132] This embodiment uses a deep learning model to intelligently screen web page content, automatically identify information related to the enterprise, and eliminate a large number of business systems that are not related to the enterprise, thereby improving the accuracy and efficiency of retrieval.

[0133] Embodiment 3: An embodiment of the present invention provides a terminal device, which includes a memory, a processor, a communication interface and a communication bus. The memory stores a program that can be run on the processor. When the processor executes the program, the Internet unregistered system monitoring method based on deep learning and multi-network space search engines in the above embodiment is implemented.

[0134] The processor may be a central processing unit, or other general-purpose processors, digital signal processors, application-specific integrated circuits, field programmable gate arrays or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or a combination of the above chips.

[0135] The memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer executable programs and units, such as the corresponding program units in the above method embodiments of the present invention. The processor executes various functional applications of the processor and works data processing by running the non-transitory software programs, instructions and modules stored in the memory, that is, the Internet unregistered system monitoring method based on deep learning and multi-network space search engines in the above embodiments is realized.

[0136] The memory may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required by at least one function; the data storage area may store data created by the processor, etc. In addition, the memory may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. The memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. One or more programs are stored in the memory, and when executed by the processor, the Internet unregistered system monitoring method based on deep learning and multi-network space search engines in the above-mentioned embodiment is executed.

[0137] Embodiment 4: The embodiment of the present invention provides a storage medium for computer-readable storage, the storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the Internet unregistered system monitoring method based on deep learning and multi-network space search engine as provided in the above method embodiment.

[0138] The storage medium may be an internal storage unit of the terminal device, such as a hard disk or memory of the terminal device. The storage medium may also be an external storage device of the terminal device, such as a plug-in hard disk, smart memory card, secure digital card, flash memory card, etc. equipped on the terminal device.

[0139] The above technical features constitute the embodiments of the present invention, which have strong adaptability and implementation effect. Non-essential technical features can be added or reduced according to actual needs to meet the requirements of different situations.

Claims

1. A monitoring device for Internet unregistered systems based on deep learning and multi-network space search engines, characterized in that Including data collection module, intelligent screening module and whitelist module; The data collection module is used to call multiple search engine APIs to retrieve data, obtain the whitelist from the whitelist module, and remove the systems in the whitelist from the search results. It accesses the system page by calling the browser kernel, collects relevant information in the page, and completes data collection. Among them, the intelligent screening module is used to extract text information from the system page, encode the text information into a vector after data preprocessing, process the encoded text vector, identify the text related to the enterprise business, predict the system feature data in the search results, and determine whether the system and the enterprise are related, so as to intelligently screen the web page content and automatically identify the information related to the enterprise; Among them, the whitelist module is used to store the registered system information and record the information irrelevant to the enterprise that was eliminated by the last deep learning-based intelligent screening module.

2. The Internet unregistered system monitoring device based on deep learning and multi-network space search engine according to claim 1 is characterized in that The data collection module includes a data retrieval component, a whitelist data processing component and a system page information collection component; the data retrieval component is used to retrieve data by calling multiple different search engine APIs; the whitelist data processing component is used to obtain the whitelist list from the whitelist module and eliminate the systems in the whitelist from the retrieval results; the system page information collection component is used to access the system page by calling the browser kernel and collect relevant information in the page.

3. The Internet unregistered system monitoring device based on deep learning and multi-network space search engine according to claim 1 or 2 is characterized in that The intelligent screening module includes a data preprocessing component and a deep learning component; the data preprocessing component is used to extract text information from the system page, and after simple preprocessing, encode the text information into a vector; the deep learning component is used to process the encoded text vector, identify text related to the enterprise business, predict the system feature data in the search results, and determine whether the system and the enterprise are related.

4. A method for monitoring unregistered Internet systems based on deep learning and multi-network space search engines, characterized in that include: Step S1: Call multiple different search engine APIs to retrieve data, obtain a whitelist from the whitelist module, and remove the systems in the whitelist from the search results, access the system page by calling the browser kernel, collect relevant information in the page, and complete data collection; Step S2: extract text information from the system page, encode the text information into a vector after data preprocessing, process the encoded text vector, identify the text related to the enterprise business, predict the system feature data in the search results, and determine whether the system and the enterprise are related, so as to intelligently screen the web page content and automatically identify information related to the enterprise.

5. The method for monitoring an unregistered Internet system based on deep learning and multiple network space search engines according to claim 4 is characterized in that In step S1, data collection specifically includes the following steps: Step S1.1: read all keywords from the keyword list, and simultaneously start the automated retrieval program of different network space search engines to retrieve all keywords; after the retrieval is completed, aggregate all the retrieval results into an intermediate file; Step S1.2: Taking the system IP, port and system title as input, calculate the MD5 value of each system; Step S1.3: Obtain the MD5 values ​​of all systems in the whitelist, compare them with the MD5 values ​​in the search results, and remove systems with the same MD5 values; Step S1.4: Start the browsing kernel, browse all remaining system pages, collect information from the pages, and then write all collected data into a file.

6. The method for monitoring an unregistered Internet system based on deep learning and multiple network space search engines according to claim 4 or 5, characterized in that In step S2, when data preprocessing is performed, the following steps are specifically included: Step S2.1: extract text information from the system page, and remove invalid words after screening; combine the screened text information with the system title to form the features of the system, and set the extracted features as D; Step S2.2: Use One-Hot encoding to encode the system features and map the characters in the text feature D. The vector representation of the mth word representing the features of system i.

7. The method for monitoring an unregistered Internet system based on deep learning and multiple network space search engines according to claim 4 or 5, characterized in that In step S2, when predicting the system characteristic data in the search results, the following steps are specifically included: Step S2.1: Transform the characteristic matrix X of system i into i The input is fed into the deep bidirectional language model pre-trained by Bert, which embeds the meaning of words and the relationship between words into the vector representation; the computational complexity of Bert is O(n 2 ), n is the length of the input feature; X i Divide into J segments according to a certain length, and input each segment into the Bert model separately; Step S2.2: Input to the linear binary classifier and predict Is each word related to the business of the enterprise? If it is relevant, it is predicted as 1, and if it is not relevant, it is predicted as 0. After the prediction is completed, the prediction result is To splice; In the above formula, and b1 represent learnable parameters, Represents a splicing operation; Step S2.3: Splice and input into the Bi-directional Long Short-Term Memory (BILSTM) model to learn the vector representation of all text features in the system X *i ; Step S2.4: The vector representation X of the entire text feature of the system *i Input to the linear binary classifier to predict whether the system is relevant to the enterprise business. If relevant, the prediction is 1, and if not, the prediction is 0; In the above formula, and b2 represent learnable parameters respectively; Step S2.5: Based on the prediction results in step S2.2, calculate the proportion of words related to the enterprise business in the entire text features of the system, and set the proportion to Step S2.6: The value is 1 and Greater than the algorithm threshold The system predicts that the website is an unregistered website of the enterprise, and annotates the words predicted by the deep learning model in the system text features according to step S2.

2.

8. The method for monitoring an unregistered Internet system based on deep learning and multiple network space search engines according to claim 4 or 5, characterized in that In step S2, before using the deep learning model to automatically filter data, it is necessary to optimize the parameters using the data that has been manually annotated in advance, which specifically includes the following steps: Step S2.1.1: Select l systems that have been registered by the enterprise and l other systems that are not related to the enterprise, perform data preprocessing, extract information from the registered systems to construct positive sample data X + , extract information from other irrelevant systems to construct negative sample data X - ; Step S2.1.2: Set X + and X - Mix them together to construct the model input data X, and annotate the words in X according to whether they are related to the business of the enterprise, and construct the word-level label vector Y w , and at the same time, the elements in X belong to X + The system is marked as 1, belonging to X - The system feature is marked as 0, and the entire text level label vector Y of the system is constructed p ; Step S2.1.3: Divide X and Y into training data sets in a ratio of 8:2 Test Dataset Step S2.1.4: Use the deep learning model f to learn X train The feature information is used to predict the word level and obtain the prediction result Y wi and system-level prediction result Y p,i ; Step S2.1.5: Use the binary cross entropy loss function to calculate the prediction results separately and the true data label Loss value between w and loss p ; Step S2.1.6: Take the loss value loss w and loss p As input, the gradient of the model parameters is calculated and the parameters of the model f are optimized using the multi-objective MTAdam optimizer f θ ’=MTAdam(loss w ,loss p ,f θ ) Step S2.1.7: Predict X using the optimized model f test , and then the prediction results and the true data label Compare and calculate the prediction accuracy of the model. If the accuracy is greater than the current optimal result, save the model parameters. Step S2.1.8: Repeat the above steps, iterate k times, gradually optimize the model parameters, and obtain the parameter values ​​under the optimal state of the model.

9. A terminal device, comprising a memory and a processor, wherein the memory stores a program that can be run on the processor, characterized in that: When the processor executes the program, the method for monitoring an unregistered Internet system based on deep learning and multi-network space search engines as described in any one of claims 4 to 8 is implemented.

10. A storage medium, characterized in that: The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the Internet unregistered system monitoring method based on deep learning and multi-network space search engines as described in any one of claims 4 to 8.