Page data detection method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention discloses a page data detection method and device, electronic equipment and a storage medium, and the method comprises the steps: operating a to-be-detected page through an automatic script, and obtaining page element data of the to-be-detected page; when the automatic script triggers an interface request, obtaining interface response data corresponding to the interface request; determining a difference set and an intersection set between the interface response data and the page element data; performing sensitive information detection on the difference set to obtain a sensitive information detection result, and performing encryption detection on the intersection to obtain an encryption detection result; and generating a data detection report according to the sensitive information detection result and the encryption detection result. According to the embodiment of the invention, the detection efficiency can be improved, and the detection comprehensiveness can be improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of network security technology, and in particular, to a page data detection method, device, electronic device and storage medium. Background Art

[0002] In today's digital age, data security and privacy protection have become the focus of attention for enterprises and users. With the popularization of Internet applications, the overexposure and unencrypted processing of sensitive information have frequently occurred, bringing huge security risks and legal risks to enterprises. In order to meet this challenge, enterprises need an efficient and automated detection method to ensure that sensitive information is fully protected during transmission and display.

[0003] In current actual security testing, traditional manual detection methods are inefficient and difficult to cover all pages and interfaces, resulting in incomplete detection of page data; existing automated detection tools have single functions and are difficult to achieve comprehensive and intelligent detection of sensitive information. Summary of the invention

[0004] The embodiments of the present application provide a page data detection method, device, electronic device and storage medium, which help to improve the comprehensiveness of page data detection and improve detection efficiency.

[0005] In order to solve the above problems, in a first aspect, an embodiment of the present application provides a page data detection method, comprising:

[0006] The page to be detected is operated by an automated script to obtain page element data of the page to be detected;

[0007] When the automation script triggers an interface request, obtaining interface response data corresponding to the interface request;

[0008] Determine a difference set and an intersection set between the interface response data and the page element data;

[0009] Performing sensitive information detection on the difference set to obtain a sensitive information detection result, and performing encryption detection on the intersection set to obtain an encryption detection result;

[0010] A data detection report is generated based on the sensitive information detection result and the encryption detection result.

[0011] In a second aspect, an embodiment of the present application provides a page data detection device, including:

[0012] A page element acquisition module is used to operate the page to be detected through an automated script to obtain page element data of the page to be detected;

[0013] An interface response acquisition module, used to acquire interface response data corresponding to the interface request when the automation script triggers the interface request;

[0014] A data comparison module, used to determine the difference and intersection between the interface response data and the page element data;

[0015] A data detection module, used to perform sensitive information detection on the difference set to obtain a sensitive information detection result, and perform encryption detection on the intersection set to obtain an encryption detection result;

[0016] The report generation module is used to generate a data detection report based on the sensitive information detection result and the encryption detection result.

[0017] In a third aspect, an embodiment of the present application further provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the page data detection method described in the embodiment of the present application when executing the computer program.

[0018] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps of the page data detection method disclosed in the embodiment of the present application are performed.

[0019] The page data detection method, device, electronic device and storage medium provided in the embodiments of the present application operate the page to be detected through an automated script to obtain page element data of the page to be detected. When the automated script triggers an interface request, the interface response data corresponding to the interface request is obtained, the difference and intersection between the interface response data and the page element data are determined, sensitive information detection is performed on the difference to obtain a sensitive information detection result, and encryption detection is performed on the intersection to obtain an encrypted detection result. A data detection report is generated based on the sensitive information detection result and the encryption detection result. Since the page to be detected can be operated through an automated script, there is no need for the user to manually operate the page, which can improve the detection efficiency. In addition, all interfaces and elements on the page can be covered by the automated script, thereby improving the comprehensiveness of the detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0021] Figure 1 is a flow chart of a page data detection method provided by an embodiment of the present application;

[0022] Figure 2 It is a structural diagram of a page data detection device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0023] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0024] Figure 1 is a flow chart of a page data detection method provided by an embodiment of the present application, such as Figure 1 As shown, the method includes: step 110 to step 150.

[0025] Step 110: operate the page to be detected through an automated script to obtain page element data of the page to be detected.

[0026] The page to be tested can be a browser page or an application page. The page to be tested can be one or more pages or all pages in the application to be tested. The automation script can simulate the user's operation behavior, and the operation behavior simulated by the automation script can include click, input, scroll and other operation behaviors.

[0027] Through the automated script, the user's operation behavior can be simulated to operate the page to be detected, such as clicking on an element in the page to be detected, inputting for an element, or scrolling the page to be detected, etc., traversing all page elements in the page to be detected, and obtaining all page elements in the page to be detected and the position of the page elements in the page to be detected by positioning, and extracting the attributes and content of the page elements, the attributes and content of the page elements can include at least one of text, links, pictures, forms, etc., converting the attributes and content of the page elements into structured data, and using the obtained position of the page elements and the attributes and content of the page elements as the page element data of the page to be detected. The extracted page element data can be stored in a database or file system for use in subsequent steps. The obtained page element data can be in JSON format, HTML fragments or other structured data.

[0028] In an exemplary embodiment, the automation script can be a Selenium script. When the page to be detected is a browser page, the user behavior can be simulated by writing a Selenium script to traverse all page elements in the page to be detected, and all elements in the page can be located and obtained through various positioning methods (such as ID, Class, XPath, CSS selectors, etc.) provided by Selenium, and the attributes and content of the page elements can be extracted. Among them, Selenium is a tool for testing Web applications. Selenium tests run directly in browsers. Selenium is a Python library for automating browser operations, which is commonly used in scenarios such as testing and crawlers of Web applications. ID is the identifier of a page element. Class is the Class attribute of a page element. Class is generally used to group elements and set the same style for elements of the same level. There will be multiple elements sharing a Class. XPath (XML Path Language) is a language used to determine the location of a part in an XML document. CSS selectors can achieve one-to-one, one-to-many or many-to-one control of elements in HTML pages.

[0029] In one embodiment of the present application, the operation of the page to be detected through an automated script to obtain page element data of the page to be detected includes: operating the page to be detected through an automated script to obtain page element data directly existing in the page to be detected; when the automated script triggers an associated page of the page to be detected, obtaining page element data in the associated page; using the page element data directly existing in the page to be detected and the page element data in the associated page as page element data of the page to be detected.

[0030] The associated pages may include hidden pages, pop-up windows or other associated pages (such as a page triggered by an element in the page to be detected and other pages associated with the associated page).

[0031] The page to be detected is operated through an automated script to obtain all page element data directly existing in the page to be detected, and when the automated script triggers the associated page of the page to be detected (for example, a page element clicked by the automated script requires opening another page), the associated page is expanded, and the associated page is operated through the automated script to obtain the page element data in the associated page, and the page element data directly existing in the page to be detected and the page element data in the associated page are used as the page element data of the page to be detected.

[0032] By using the page element data directly existing in the page to be detected and the page element data in the associated pages as the page element data of the page to be detected, more comprehensive page element data can be extracted, further improving the comprehensiveness of page detection.

[0033] Step 120, when the automation script triggers an interface request, obtain interface response data corresponding to the interface request.

[0034] A proxy server can be pre-configured. When an automated script triggers an interface request, all interface requests and interface response data corresponding to the interface requests are intercepted and recorded by the proxy server. The proxy server can be, for example, BrowserMob Proxy. Each interface request can include an HTTP request or an HTTPS request. The interface response data can include a request URL (Uniform Resource Locator), request parameters, response status code, response data, etc.

[0035] In an exemplary embodiment, BrowserMob Proxy can be configured as a proxy server for the browser to intercept and record all HTTP / HTTPS requests and responses; start the Selenium script, automate the script to access the page to be detected, and record all interface request and response data through BrowserMob Proxy; capture all interface request and response data involved in the page to be detected, including URL, request parameters, response status code, response data, etc.; parse the interface response data, extract the returned data content in JSON, XML, etc. format as the required interface response data, and the extracted interface response data can be structured and stored in the form of key-value; store the interface response data in a database or file system for use in subsequent steps. The key in the interface response data represents the data identifier, and the value represents the data value. The data value can be, for example, a mobile phone number, an ID number, etc.

[0036] Step 130: Determine the difference and intersection between the interface response data and the page element data.

[0037] Compare the interface response data and the page element data, and take the data that exists in the interface response data but not in the page element data as the difference set, and take the data that exists in the interface response data and also in the page element data as the intersection set, that is, the difference set is the set of data returned by the backend (such as the backend server) but not used by the frontend (such as the browser), and the intersection set is the set of data returned by the backend and used by the frontend.

[0038] In one embodiment of the present application, the interface response data includes a data identifier and a data value, and the page element data includes a page element identifier;

[0039] The determining of the difference and intersection between the interface response data and the page element data includes: comparing each data identifier in the interface response data with each page element identifier of the page element data, determining the interface response data corresponding to the data identifier that does not exist in the page element identifier as the difference, and determining the interface response data corresponding to the data identifier that exists in the page element identifier as the intersection.

[0040] The interface response data is represented in the form of key-value, including data identifier and data value. The data value is specific data, such as a mobile phone number, ID number, etc. The page element identifier is the identifier of the page element in the page to be detected. If a page element is a displayed data, the page element identifier of the page element is the same as the data identifier of the displayed data.

[0041] Compare each data identifier in the interface response data with each page element identifier in the page element data to determine whether each data identifier is the same as the page element identifier. If a data identifier in the interface response data is the same as a page element identifier in the page element data, then the data identifier exists in the page element data identifier. If a data identifier in the interface response data is different from all page element identifiers in the page element data, then the data identifier does not exist in the page element data identifier. Count all data identifiers that do not exist in the page element identifier, and determine the set consisting of the interface response data corresponding to each of the data identifiers as a difference set; and count all data identifiers that exist in the page element identifier, and determine the set consisting of the interface response data corresponding to each of the data identifiers as an intersection set.

[0042] By comparing each data identifier in the interface response data with each page element identifier in the page element data, the difference set and the intersection set can be accurately determined, and then different sensitive information detections can be performed on the difference set and the intersection set.

[0043] Step 140: Perform sensitive information detection on the difference set to obtain a sensitive information detection result, and perform encryption detection on the intersection set to obtain an encryption detection result.

[0044] Each data in the difference set can be compared with each sensitive information in the pre-configured sensitive information standard library to determine whether each data in the difference set exists in the sensitive information standard library. If a data exists in the sensitive information standard library, the data is determined to be sensitive information, all sensitive information in the difference set is obtained, and the sensitive information detection result is obtained.

[0045] For the data in the intersection, determine whether each data is encrypted, obtain all the data in the intersection that are not encrypted, and obtain the encryption detection result.

[0046] In one embodiment of the present application, the sensitive information detection is performed on the difference set to obtain the sensitive information detection result, including: comparing each data in the difference set with each sensitive information in the sensitive information standard library, determining the first sensitive information in the difference set and the security level corresponding to the first sensitive information, and using the first sensitive information and the security level corresponding to the first sensitive information as the sensitive information detection result.

[0047] Each piece of sensitive information stored in the sensitive information standard library includes sensitive information type, historical data and security level. Sensitive information types may include personal identity information (such as mobile phone number, ID card number, etc.), payment information (such as bank card number, etc.), medical information (such as medical insurance account number, etc.), etc.

[0048] Each data in the difference set is compared with each sensitive information in the sensitive information standard library. If a data in the difference set hits a sensitive information in the sensitive information standard library, the data is determined to be the first sensitive information, and the security level corresponding to the hit sensitive information in the sensitive information standard library is used as the security level corresponding to the first sensitive information. In this way, the security level corresponding to all the first sensitive information in the difference set and each first sensitive information is determined to obtain the sensitive information detection result of the difference set.

[0049] By comparing the data in the difference set with the sensitive information in the sensitive information standard library, the first sensitive information in the difference set can be accurately determined, and the security level corresponding to the first sensitive information can be determined, so as to facilitate the subsequent generation of corresponding improvement suggestions based on the first sensitive information and the security level.

[0050] In one embodiment of the present application, the encryption detection is performed on the intersection to obtain an encryption detection result, including: when displaying the data in the intersection, determining whether the displayed data is hidden and obtaining a hidden processing detection result; comparing the data in the intersection that has not been encrypted with each sensitive information in a sensitive information standard library to determine the second sensitive information in the data that has not been encrypted and the security level corresponding to the second sensitive information; and using the hidden processing detection result, the second sensitive information and the security level corresponding to the second sensitive information as the encryption detection result.

[0051] When displaying data in the intersection, determine whether the displayed data has been hidden. For example, the hiding method may be to use '**' to replace part of the information therein, and determine the data that has not been hidden from the intersection as the hidden detection result.

[0052] An encryption check is performed on each data in the intersection to determine the data that has not been encrypted, and the data that has not been encrypted is compared with each sensitive information in the sensitive information standard library. If one data in the data that has not been encrypted hits a sensitive information in the sensitive information standard library, the data is determined to be second sensitive information, and the security level corresponding to the hit sensitive information in the sensitive information standard library is used as the security level corresponding to the second sensitive information. In this way, all the second sensitive information in the data that has not been encrypted and the security level corresponding to each second sensitive information are determined.

[0053] The hidden processing detection result, the second sensitive information and the security level corresponding to the second sensitive information are used as the encryption detection result.

[0054] By performing hidden processing on the data in the intersection and determining the second sensitive information and the corresponding security level from the data that has not been encrypted, it is possible to detect whether the data is hidden and encrypted, thereby facilitating the subsequent generation of corresponding improvement suggestions based on the encryption detection results to improve the security of sensitive information.

[0055] Step 150: Generate a data detection report based on the sensitive information detection result and the encryption detection result.

[0056] Summarize the sensitive information detection results and encryption detection results to generate a data detection report. When generating a data detection report, you can use BeautifulReport, a Python library for generating beautiful test reports. It can output test results in multiple formats, including HTML, PDF, and JUnit XML. The format of the generated data detection report is not limited, for example, it can be HTML, PDF, etc., which is easy to view and share.

[0057] The report content of the data detection report may include: detection overview, sensitive information detection results and encryption detection results. The detection overview may include basic information such as detection scope, detection time, detection tools, etc. Sensitive information detection results may include the type, location, security level, etc. of sensitive information hit. Encryption detection results may include unencrypted data items, locations, security levels, etc.

[0058] In one embodiment of the present application, generating a data detection report based on the sensitive information detection results and the encryption detection results includes: generating data improvement suggestions based on the sensitive information detection results and the encryption detection results; generating a data detection report including the sensitive information detection results, the encryption detection results and the data improvement suggestions.

[0059] Since the sensitive information detection results are detected from the difference set, and the data of the difference set will not be displayed on the front end, the returned data has the risk of sensitive information leakage. In order to avoid this risk, based on the sensitive information detection results, data improvement suggestions can be given that do not need to return the data in the sensitive information detection results (i.e., the first sensitive information). The encryption detection results are detected from the intersection, and the data in the intersection needs to be displayed. For the hidden processing detection results in the encryption detection results, if a certain data has not been hidden, the data improvement suggestion given can be that the data needs to be hidden; for the data that has not been encrypted in the encryption detection results, the data improvement suggestion given can be to recommend that the data be encrypted.

[0060] In addition to the above-mentioned detection overview, sensitive information detection results and encryption detection results, the report content of the generated data detection report can also include data improvement suggestions, so that developers or testers can handle the sensitive data in the page to be detected accordingly according to the data improvement suggestions.

[0061] In an exemplary embodiment, when generating a data detection report, a report template preset in advance can be used to automatically generate a detailed data detection report through a large language model based on the sensitive information detection results and encryption detection results. The data detection report can include the type, location, security level, etc. of the detected sensitive information; provide repair suggestions and best practices to help users handle and protect sensitive information in a timely manner.

[0062] The page data detection method provided in the embodiment of the present application operates the page to be detected through an automated script to obtain page element data of the page to be detected. When the automated script triggers an interface request, the interface response data corresponding to the interface request is obtained, the difference and intersection between the interface response data and the page element data are determined, sensitive information detection is performed on the difference to obtain a sensitive information detection result, and encryption detection is performed on the intersection to obtain an encrypted detection result. A data detection report is generated based on the sensitive information detection result and the encryption detection result. Since the page to be detected can be operated through an automated script, there is no need for the user to manually operate the page, which can improve the detection efficiency. Moreover, all interfaces and elements on the page can be covered by the automated script, thereby improving the comprehensiveness of the detection.

[0063] On the basis of the above technical solution, before the operation on the page to be detected is performed through the automated script, it also includes: parsing the HTML structure of the page to be detected, obtaining the page elements in the page to be detected and the attributes corresponding to the page elements; generating a simulation operation sequence of the page to be detected according to the page elements and the attributes through the first large language model; generating an automated script corresponding to the simulation operation sequence according to a predefined template through the first large language model.

[0064] The HTML (Hyper Text Markup Language) structure of the page to be tested is obtained through an automated tool (such as Selenium), the HTML structure is parsed, and various page elements (such as buttons, input boxes, links, etc.) and their attributes (such as ID, class name, tag name, etc.) in the page are extracted.

[0065] By pre-training and fine-tuning the large language model, common web page interaction patterns and user operation paths are learned to obtain the first large language model. The page structure of the page to be detected is analyzed through the first large language model, that is, the page elements and attributes of the page to be detected are analyzed to infer the possible operation path of the user. For example, key operation points such as the login button and the form submission button are identified. The first large language model generates a possible user operation sequence based on the attributes and context of the page elements as a simulated operation sequence of the page to be detected. The simulated operation sequence may include operations such as clicking, inputting, and scrolling.

[0066] The first language model can generate script code corresponding to the simulated operation sequence according to the predefined template (for example, it can be Python Selenium code) to obtain an automation script. The generated automation script may include steps such as page loading, element positioning, and operation execution. Among them, the predefined template is a template that includes script codes corresponding to different user operations.

[0067] After the automation script is generated by the first language model, the automation script can be executed to operate the page to be detected, and the execution result of the automation script can be fed back to the first language model, so that the first language model can automatically adjust the automation script according to the dynamic changes of the page to be detected. By simulating multiple operations, it is verified whether the generated automation script can run stably, and necessary adjustments and optimizations are made based on the verification results, and finally an automation script that conforms to the user's operation behavior and can cover all operations in the page to be detected is generated.

[0068] By automatically parsing the HTML structure of the page to be detected, the page elements and attributes in the page to be detected are obtained, and then a simulation operation sequence of the page to be detected can be automatically generated based on the first large language model, and an automation script can be generated based on the simulation operation sequence, thereby realizing the automatic generation of automation scripts. Moreover, the automation script generated by the large language model can cover all page elements and corresponding operations in the page to be detected, thereby improving the comprehensiveness of the detection scope.

[0069] On the basis of the above technical solution, the determination of the difference and intersection between the interface response data and the page element data includes: preprocessing the interface response data and the page element data to convert the interface response data and the page element data into data in a target format; performing a semantic comparison on the preprocessed interface response data and the page element data through a second largest language model to determine the difference and intersection between the preprocessed interface response data and the page element data.

[0070] By executing automated scripts to operate the pages to be detected, page element data and interface response data are collected. These data can be in JSON format, HTML fragments, or other structured data. The collected page element data and interface response data are preprocessed, including data cleaning, format conversion, etc., to convert the interface response data and page element data into data in a unified target format, ensuring that the interface response data and page element data are in a unified format before comparison.

[0071] The interface response data and page element data are input into the second largest language model, and the semantic understanding ability of the large language model is used to compare the page element data and the interface response data. The second largest language model can automatically identify the key fields and content in the page element data and the interface response data, and perform a comparison at the semantic level. The second largest language model can automatically calculate the difference and intersection between the page element data and the interface response data based on the comparison results.

[0072] The second largest language model is used to perform semantic comparison on the interface response data and the page element data to determine the difference and intersection. This allows the comparison logic of different interfaces and page elements to be automatically processed based on the contextual understanding capability of the large language model, without the need to write specific processing logic for each interface and page element.

[0073] On the basis of the above technical solution, the sensitive information detection is performed on the difference set to obtain a sensitive information detection result, and the encryption detection is performed on the intersection to obtain an encrypted detection result, including: identifying the sensitive information in the difference set through a third largest language model to obtain a sensitive information detection result, and encrypting the intersection through the third largest language model to obtain an encrypted detection result.

[0074] The large language model is pre-trained and fine-tuned to obtain the third language model, which can identify various forms of sensitive information (such as personal identity information, payment information, medical information, etc.); in the obtained difference sets and intersection sets, the third language model can automatically detect and mark possible sensitive information. The difference sets and intersection sets between the interface response data and the page element data are input into the trained third language model, and the data in the difference set is compared with the sensitive information in the sensitive information standard library through the third language model to detect whether there is sensitive information in the difference set and obtain the sensitive information detection result; the data in the intersection set is encrypted and detected through the third language model to detect whether the data in the intersection has been hidden and encrypted, and obtain the encryption detection result.

[0075] The third language model can cover more types of sensitive information through continuous training data updates. Training data can include public sensitive information samples, user-provided annotated data, etc. Based on the new training data, the model can automatically update and optimize the detection rules to maintain efficient sensitive information detection capabilities.

[0076] On the basis of the above technical solution, after generating the data detection report according to the sensitive information detection result and the encryption detection result, it also includes: receiving feedback results of the target personnel on the data detection report, the feedback results including the correction results of the sensitive information detection results; according to the correction results, updating the third language model.

[0077] After the data detection report is generated, the data detection report can be sent to the target person, who can verify the data detection report and give feedback including the correction results corresponding to the sensitive information detection results. The third language model can be updated based on the correction results of the target person.

[0078] The large language model can continuously improve the accuracy and comprehensiveness of sensitive information detection through continuous learning and feedback mechanisms. Through user feedback and actual detection results, the large language model can automatically adjust and optimize detection strategies to adapt to new data formats and threat intelligence.

[0079] The third language model and the second language model can be different large language models or the same large language model. When the third language model and the second language model are the same large language model, the large language model needs to be trained before the trained large language model is used. The training process of the large language model may include the following steps:

[0080] Data collection: Collect a large amount of page element data, interface response data and sensitive information samples to build a data set. The collected data includes various types of page elements, data formats returned by the interface, and common sensitive information types (such as personal information, financial data, etc.);

[0081] Data preprocessing: Clean and preprocess the collected data, including removing noise data, standardizing data formats, labeling sensitive information, etc., and divide the data set consisting of the preprocessed data into a training data set and a validation data set.

[0082] Model training: Use the preprocessed training data set to train the generative pre-trained model. During the training process, adjust the model parameters and hyperparameters to improve the accuracy and generalization ability of the model;

[0083] Model validation: Use the validation data set to validate the trained generative pre-trained model and evaluate the performance of the model. The evaluation indicators may include accuracy, recall, and F1 Score.

[0084] Model optimization: Based on the verification results, the model is further optimized. Multiple iterations of training and verification may be required until the performance of the generative pre-trained model meets the expected requirements and a trained generative pre-trained model is obtained.

[0085] By using a trained generative pre-trained model to determine the difference and intersection between interface response data and page element data, as well as performing sensitive information detection on the difference and encryption detection on the intersection, the efficiency and accuracy of data detection can be significantly improved when the data volume is large.

[0086] On the basis of the above technical solution, after generating the data detection report according to the sensitive information detection result and the encryption detection result, it also includes: storing the data detection report, and generating a viewing link for the data detection report according to the storage location of the data detection report; and pushing the viewing link to the target person.

[0087] The data detection report is stored in a database or file system, a viewing link for the data detection report is generated according to the storage location of the data detection report, and the viewing link is pushed to the target person. When a viewing request is received from the target person by clicking the viewing link, the data detection report is obtained from the storage location of the data detection report and returned to the target person, so that the target person can view the data detection report, verify the data detection report, and obtain feedback from the target person after verification. The feedback may include confirming the problem, making suggestions for improvement, etc.

[0088] When pushing the viewing link to the target person, you can push it through a push message. The push message content may include a test overview, a summary of the data test results, and a viewing link, etc., so that the relevant person can quickly understand the test situation. When pushing messages, for example, you can use Lark Robot to push.

[0089] After obtaining the feedback content after verification by the target personnel, the generative pre-training model can be further optimized based on the feedback content to improve the performance of the model.

[0090] By sending the viewing link of the data detection report to the target person, the target person can understand the results of the data detection and can verify the data detection results. It is also convenient for the target person to perform corresponding processing on the page to be detected, which can ensure that the data detection results are communicated and processed in a timely manner to improve the security of sensitive information.

[0091] Based on the above technical solutions, you can configure an automated pipeline through Jenkins to regularly execute Selenium scripts and BrowserMob Proxy data capture to ensure the continuous operation of the detection process; you can deploy the detection Selenium scripts and generative pre-trained models to the Jenkins environment, configure scheduled tasks or trigger conditions to automatically execute page data detection tasks.

[0092] The page data detection method provided in the embodiment of the present application can be triggered regularly by a scheduled task, or, when the page of the application to be tested meets the trigger condition (for example, the page of the application to be tested is updated), the page is used as the page to be detected and data detection is performed on the page to be detected.

[0093] The embodiment of the present application is based on the SPADE intelligent proxy framework, integrating GPT, Selenium, BrowserMobProxy, BeautifulReport, Jenkins, and Feishu Robot tools, and implements a method for automatically detecting excessive exposure and unencrypted sensitive information:

[0094] First, click on the page through Selenium's self-developed framework script to obtain the page elements, and use BrowserMobProxy to obtain the interfaces involved in the page to be detected and the output parameters of the interfaces (response data); then give the page element data and the output parameter data of the interface to GPT, let it extract key features of the page elements and compare them with the output parameters of the interface, and get two sets of data, one is the difference set (that is, part of the data returned by the backend but not used by the frontend), and the other is the intersection set (that is, the data returned by the backend and also used by the frontend); secondly, let GPT compare the data in the difference set with the preset sensitive information standard library. If a hit is found, the security level of the data will be determined, and the intersection set will be compared with the default sensitive information standard library. Centralized data detection is used to determine whether hidden processing (such as using '**' to hide) and encryption are performed. If not encrypted, the security level of the data is also determined; then the above steps are repeated, and all hidden pages and pop-up windows are clicked through the Selenium script to obtain page elements and their corresponding interface data, and then the detection is continued; finally, a data detection report is generated for all the detection records obtained, and pushed to the corresponding target personnel through push messages. The target personnel will verify the detection results and feedback the results to GPT for further training; the entire process is connected in series through intelligent agents and data interaction between agents, and the project is deployed through Jenkins for continuous integration. Among them, Jenkins is an open source software project, a continuous integration tool developed based on Java, used to monitor continuous and repetitive work, and aims to provide an open and easy-to-use software platform so that software projects can be continuously integrated.

[0095] The embodiments of the present application determine and detect differences and intersections based on the acquisition of page element data, the acquisition of interface response data, and a generative pre-trained model, and can generate a data detection report, thereby achieving efficient, intelligent, and comprehensive sensitive information detection, helping enterprises or users to promptly discover and deal with data security risks, and protecting user privacy and data security; its advantages of efficient automation, comprehensive coverage, intelligent data comparison, sensitive information detection, encryption verification, real-time reporting and notification, continuous integration and optimization, etc., give it a significant advantage in the field of data security and privacy protection.

[0096] Among them, efficient automation is reflected in the automatic acquisition of page element and interface data through Selenium scripts and BrowserMob Proxy, which greatly improves the detection efficiency and reduces the time and cost of manual operations; comprehensive coverage is reflected in the ability to automatically traverse all page elements, including related pages (hidden pages and pop-ups), to ensure that the detection scope is comprehensive and there are no blind spots; intelligent data comparison is reflected in the use of generative pre-trained models for intelligent data comparison and feature extraction, which can accurately identify and compare page element data with the response data returned by the interface, reducing misjudgments and missed judgments; sensitive information detection is reflected in the automatic detection of sensitive information in data returned by the back end but not used by the front end through the preset sensitive information standard library, so as to timely discover potential data leakage risks; real-time reporting and notification is reflected in the generation of detailed data detection reports, and pushing them to relevant personnel in real time through push messages to ensure that the detection results are communicated and processed in a timely manner.

[0097] Figure 2 is a schematic diagram of the structure of a page data detection device provided in an embodiment of the present application, such as Figure 2 As shown, the device comprises:

[0098] The page element acquisition module 210 is used to operate the page to be detected through an automated script to obtain page element data of the page to be detected;

[0099] The interface response acquisition module 220 is used to acquire the interface response data corresponding to the interface request when the automation script triggers the interface request;

[0100] A data comparison module 230, used to determine the difference and intersection between the interface response data and the page element data;

[0101] The data detection module 240 is used to perform sensitive information detection on the difference set to obtain a sensitive information detection result, and perform encryption detection on the intersection set to obtain an encryption detection result;

[0102] The report generation module 250 is used to generate a data detection report according to the sensitive information detection result and the encryption detection result.

[0103] Optionally, the device further comprises:

[0104] A page parsing module is used to parse the HTML structure of the page to be detected, and obtain page elements in the page to be detected and attributes corresponding to the page elements;

[0105] An operation sequence generating module, used to generate a simulation operation sequence of the page to be detected according to the page elements and the attributes through a first large language model;

[0106] A simulation script generation module is used to generate an automation script corresponding to the simulation operation sequence according to a predefined template using the first language model.

[0107] Optionally, the interface response data includes a data identifier and a data value, and the page element data includes a page element identifier;

[0108] The data comparison module is specifically used for:

[0109] Each data identifier in the interface response data is compared with each page element identifier of the page element data, and the interface response data corresponding to the data identifier that does not exist in the page element identifier is determined as the difference set, and the interface response data corresponding to the data identifier that exists in the page element identifier is determined as the intersection.

[0110] Optionally, the page element acquisition module includes:

[0111] A direct element acquisition unit, used to operate the page to be detected through an automated script to obtain page element data directly existing in the page to be detected;

[0112] An associated page element acquisition unit, configured to acquire page element data in an associated page of the page to be detected when the automated script triggers the associated page of the page to be detected;

[0113] The page element determination unit is used to use the page element data directly existing in the page to be detected and the page element data in the associated page as the page element data of the page to be detected.

[0114] Optionally, the report generation module includes:

[0115] A suggestion generating unit, configured to generate a data improvement suggestion according to the sensitive information detection result and the encryption detection result;

[0116] A report generating unit is used to generate a data detection report including the sensitive information detection result, the encryption detection result and the data improvement suggestion.

[0117] Optionally, the data detection module includes:

[0118] The sensitive information detection unit is used to compare each data in the difference set with each sensitive information in the sensitive information standard library, determine the first sensitive information in the difference set and the security level corresponding to the first sensitive information, and use the first sensitive information and the security level corresponding to the first sensitive information as the sensitive information detection result.

[0119] Optionally, the data detection module includes an encryption detection unit, and the encryption detection unit is used to:

[0120] When displaying the data in the intersection, determining whether the displayed data is subjected to a hidden processing, and obtaining a hidden processing detection result;

[0121] Compare the unencrypted data in the intersection with each sensitive information in the sensitive information standard library to determine the second sensitive information in the unencrypted data and the security level corresponding to the second sensitive information;

[0122] The hidden processing detection result, the second sensitive information and the security level corresponding to the second sensitive information are used as the encryption detection result.

[0123] Optionally, the device further comprises:

[0124] A viewing link generating module, used for storing the data detection report and generating a viewing link for the data detection report according to the storage location of the data detection report;

[0125] The viewing link push module is used to push the viewing link to the target person.

[0126] Optionally, the data comparison module is specifically used for:

[0127] Preprocessing the interface response data and the page element data to convert the interface response data and the page element data into data in a target format;

[0128] The preprocessed interface response data and the page element data are semantically compared through the second largest language model to determine the difference set and the intersection set between the preprocessed interface response data and the page element data.

[0129] Optionally, the data detection module is specifically used for:

[0130] The sensitive information in the difference set is identified by using a third language model to obtain a sensitive information detection result, and the intersection set is encrypted and detected by using the third language model to obtain an encrypted detection result.

[0131] Optionally, the device further comprises:

[0132] A feedback result receiving module, used to receive the feedback result of the target person on the data detection report, wherein the feedback result includes the correction result of the sensitive information detection result;

[0133] A model updating module is used to update the third language model according to the correction result.

[0134] The page data detection device provided in the embodiment of the present application is used to implement the various steps of the page data detection method described in the embodiment of the present application. The specific implementation methods of each module of the device refer to the corresponding steps and will not be repeated here.

[0135] The page data detection device provided in the embodiment of the present application operates the page to be detected through an automated script to obtain page element data of the page to be detected. When the automated script triggers an interface request, the interface response data corresponding to the interface request is obtained, the difference and intersection between the interface response data and the page element data are determined, sensitive information detection is performed on the difference to obtain a sensitive information detection result, and encryption detection is performed on the intersection to obtain an encrypted detection result. A data detection report is generated based on the sensitive information detection result and the encryption detection result. Since the page to be detected can be operated through an automated script, there is no need for the user to manually operate the page, which can improve the detection efficiency. Moreover, all interfaces and elements on the page can be covered by the automated script, thereby improving the comprehensiveness of the detection.

[0136] Accordingly, an embodiment of the present application further provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the page data detection method as described in the embodiment of the present application when executing the computer program. The electronic device may be a PC, a mobile terminal, a personal digital assistant, a tablet computer, etc.

[0137] The embodiment of the present application also provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps of the page data detection method described in the embodiment of the present application are implemented.

[0138] Each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0139] The above is a detailed introduction to a page data detection method, device, electronic device and storage medium provided in an embodiment of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

[0140] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

Claims

1. A page data detection method, characterized in that: include: The page to be detected is operated by an automated script to obtain page element data of the page to be detected; When the automation script triggers an interface request, obtaining interface response data corresponding to the interface request; Determine a difference set and an intersection set between the interface response data and the page element data; Performing sensitive information detection on the difference set to obtain a sensitive information detection result, and performing encryption detection on the intersection set to obtain an encryption detection result; A data detection report is generated based on the sensitive information detection result and the encryption detection result.

2. The method according to claim 1, characterized in that: Before operating the page to be detected by the automated script, the following steps are also included: Parsing the HTML structure of the page to be detected, obtaining page elements in the page to be detected and attributes corresponding to the page elements; Generate a simulation operation sequence of the page to be detected according to the page elements and the attributes by using a first large language model; An automation script corresponding to the simulation operation sequence is generated according to a predefined template through the first language model.

3. The method according to claim 1, characterized in that The interface response data includes a data identifier and a data value, and the page element data includes a page element identifier; The determining of the difference set and the intersection set between the interface response data and the page element data includes: Each data identifier in the interface response data is compared with each page element identifier of the page element data, and the interface response data corresponding to the data identifier that does not exist in the page element identifier is determined as the difference set, and the interface response data corresponding to the data identifier that exists in the page element identifier is determined as the intersection.

4. The method according to claim 1, characterized in that: The step of operating the page to be detected by using an automated script to obtain page element data of the page to be detected includes: The page to be detected is operated by an automated script to obtain page element data directly existing in the page to be detected; When the automation script triggers a page associated with the page to be detected, obtaining page element data in the associated page; The page element data directly existing in the page to be detected and the page element data in the associated page are used as the page element data of the page to be detected.

5. The method according to claim 1, characterized in that The generating a data detection report according to the sensitive information detection result and the encryption detection result includes: Generate data improvement suggestions based on the sensitive information detection result and the encryption detection result; Generate a data detection report including the sensitive information detection result, the encryption detection result and the data improvement suggestion.

6. The method according to any one of claims 1 to 5, characterized in that: The performing sensitive information detection on the difference set to obtain a sensitive information detection result includes: Each data in the difference set is compared with each sensitive information in the sensitive information standard library, the first sensitive information in the difference set and the security level corresponding to the first sensitive information are determined, and the first sensitive information and the security level corresponding to the first sensitive information are used as the sensitive information detection result.

7. The method according to any one of claims 1 to 5, characterized in that: The performing encryption detection on the intersection to obtain an encryption detection result includes: When displaying the data in the intersection, determining whether the displayed data is subjected to a hidden processing, and obtaining a hidden processing detection result; Compare the unencrypted data in the intersection with each sensitive information in the sensitive information standard library to determine the second sensitive information in the unencrypted data and the security level corresponding to the second sensitive information; The hidden processing detection result, the second sensitive information and the security level corresponding to the second sensitive information are used as the encryption detection result.

8. The method according to any one of claims 1 to 5, characterized in that: After generating a data detection report according to the sensitive information detection result and the encryption detection result, the method further includes: Storing the data detection report, and generating a viewing link for the data detection report according to the storage location of the data detection report; The viewing link is pushed to the target person.

9. The method according to any one of claims 1 to 5, characterized in that: The determining of the difference set and the intersection set between the interface response data and the page element data includes: Preprocessing the interface response data and the page element data to convert the interface response data and the page element data into data in a target format; The preprocessed interface response data and the page element data are semantically compared through the second largest language model to determine the difference set and the intersection set between the preprocessed interface response data and the page element data.

10. The method according to any one of claims 1 to 5, characterized in that: The performing sensitive information detection on the difference set to obtain a sensitive information detection result, and performing encryption detection on the intersection set to obtain an encryption detection result, includes: The sensitive information in the difference set is identified by using a third language model to obtain a sensitive information detection result, and the intersection set is encrypted and detected by using the third language model to obtain an encrypted detection result.

11. The method according to claim 10, characterized in that After generating a data detection report according to the sensitive information detection result and the encryption detection result, the method further includes: Receiving feedback from the target person on the data detection report, wherein the feedback includes a correction result of the sensitive information detection result; The third language model is updated according to the correction result.

12. A page data detection device, characterized in that: include: A page element acquisition module is used to operate the page to be detected through an automated script to obtain page element data of the page to be detected; An interface response acquisition module, used to acquire interface response data corresponding to the interface request when the automation script triggers the interface request; A data comparison module, used to determine the difference and intersection between the interface response data and the page element data; A data detection module, used to perform sensitive information detection on the difference set to obtain a sensitive information detection result, and perform encryption detection on the intersection set to obtain an encryption detection result; A report generation module is used to generate a data detection report based on the sensitive information detection result and the encryption detection result.

13. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the page data detection method according to any one of claims 1 to 11 is implemented.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the page data detection method described in any one of claims 1 to 11 are implemented.

Images