Webpage data collection method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a webpage data collection method and device, electronic equipment and a storage medium, and belongs to the technical field of webpage data collection. The method comprises the steps that firstly, a target webpage is obtained, and then in response to access operation on the target webpage, the network quality state, the data transmission quantity and original webpage operation of the target webpage are obtained, and the original webpage operation comprises webpage sub-operation. And determining the webpage collection frequency according to at least one of the network quality state, the data transmission quantity and the webpage sub-operation. And finally, collecting the target webpage according to the webpage collection frequency. According to the embodiment of the invention, the webpage optimization efficiency can be improved.

Description

Technical Field

[0001] The present application relates to the technical field of web page data collection, and in particular to a web page data collection method and device, electronic equipment and storage medium. Background Art

[0002] In related technologies, web page analysis tools or screen recording technology based on Web Real-Time Communication (WebRTC) are used to collect behavioral information such as page views, bounce rates, and user stay time to help understand user needs and optimize web page design in a timely manner. However, web page analysis tools and screen recording technology will collect all behavioral data of web pages during the operation phase, resulting in a large amount of time cost when analyzing behavioral data, which reduces the efficiency of web page optimization. Summary of the invention

[0003] The main purpose of the embodiments of the present application is to provide a web page data collection method and device, an electronic device and a storage medium, aiming to improve the efficiency of web page optimization.

[0004] To achieve the above-mentioned purpose, a first aspect of an embodiment of the present application proposes a webpage data collection method, the method comprising:

[0005] Get the target web page;

[0006] In response to the access operation to the target webpage, the network quality status, data transmission volume and original webpage operation of the target webpage are obtained; wherein the original webpage operation includes webpage sub-operations;

[0007] determining a web page acquisition frequency according to at least one of the network quality status, the data transmission volume, and the web page sub-operation;

[0008] The target web page is collected according to the web page collection frequency.

[0009] In some embodiments, after collecting the target web page according to the web page collection frequency, the method further includes:

[0010] Performing webpage anomaly detection on the target webpage to obtain an anomaly timestamp;

[0011] Acquiring original web page data obtained by collecting the target web page;

[0012] Marking the original web page data according to the abnormal timestamp to obtain target web page data;

[0013] The target web page data is stored in a preset database.

[0014] In some embodiments, the acquiring of original webpage data collected from the target webpage includes:

[0015] Acquire initial web page data obtained by collecting the target web page; wherein the initial web page data includes web page image data and operation log data;

[0016] Performing sensitive area detection on the webpage image data to obtain sensitive areas;

[0017] Performing image masking on the sensitive area to obtain a target web page image;

[0018] Obtaining the confidentiality level of the operation log data;

[0019] Encrypting the operation log data according to the confidentiality level to obtain encrypted log data;

[0020] The original web page data is obtained according to the target web page image and the encrypted log data.

[0021] In some embodiments, storing the target webpage data in a preset database includes:

[0022] Acquire the data storage capacity of the target webpage data;

[0023] If the data storage capacity is greater than or equal to the preset storage capacity, slicing the target web page data to obtain sliced ​​web page data;

[0024] Compressing the fragmented web page data to obtain compressed web page data;

[0025] The compressed web page data is stored in the preset database.

[0026] In some embodiments, storing the target webpage data in a preset database includes:

[0027] Acquire a first network connection status; the first network connection status is used to indicate that the network is connected or not connected;

[0028] If the first network connection state is network connection, storing the target web page data in the preset database;

[0029] If the first network connection state is that the network is not connected, the target web page data is stored in a local cache, the current network connection state is acquired, and a second network connection state is obtained;

[0030] The target web page data is extracted from the local cache according to the second network connection state to obtain intermediate web page data, and the intermediate web page data is stored in the preset database.

[0031] In some embodiments, the web page acquisition frequency includes an operation page acquisition frequency and an operation data acquisition frequency; and determining the web page acquisition frequency according to at least one of the network quality status, the data transmission volume, and the web page sub-operation includes:

[0032] Determining the operation page collection frequency according to at least one of the network quality status and the data transmission volume;

[0033] Determining the operation data collection frequency according to the webpage sub-operation;

[0034] The web page collection frequency is determined according to the operation page collection frequency and the operation data collection frequency.

[0035] In some embodiments, determining the operation data collection frequency according to the webpage sub-operation includes:

[0036] Obtaining the operation type of the webpage sub-operation;

[0037] The preset acquisition frequency table is queried according to the operation type to obtain the operation data acquisition frequency; wherein the preset acquisition frequency table is used to characterize the mapping relationship between the operation type and the acquisition frequency.

[0038] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application provides a webpage data collection device, the device comprising:

[0039] A first acquisition module, used to acquire a target web page;

[0040] A second acquisition module, configured to acquire the network quality status, data transmission volume, and original web page operation of the target web page in response to the access operation to the target web page; wherein the original web page operation includes web page sub-operations;

[0041] A collection frequency module, used to determine a web page collection frequency according to at least one of the network quality status, the data transmission volume and the web page sub-operation;

[0042] The data collection module is used to collect the target web page according to the web page collection frequency.

[0043] To achieve the above-mentioned purpose, the third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the method described in the first aspect is implemented.

[0044] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in the first aspect is implemented.

[0045] The web page data collection method and device, electronic device and storage medium proposed in this application can first clarify the scope and target of data collection by acquiring the target web page. Then, in response to the access operation, further obtain the network quality status, data transmission volume and the user's original web page operation to ensure that the data collection process can adapt to the actual network conditions and visitor operations, and then dynamically adjust the collection strategy according to the current network conditions and operation requirements. Finally, the web page data is collected in real time according to the web page collection frequency determined in the above steps. Compared with the solution of collecting all data in full in the prior art, the method of this embodiment can adaptively adjust the web page collection frequency according to different environmental conditions during the collection process, effectively reducing the amount of collected data, thereby avoiding the waste of storage resources and reducing data complexity, thereby reducing the time and complexity required for subsequent analysis of collected data and formulation of web page optimization strategies, and ultimately improving the efficiency of web page optimization. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 is a flow chart of a web page data collection method provided by an embodiment of the present application;

[0047] Figure 2 yes Figure 1 Flow chart of step S103 in FIG.

[0048] Figure 3 yes Figure 2 Flow chart of step S202 in FIG.

[0049] Figure 4 is another flow chart of the webpage data collection method provided by an embodiment of the present application;

[0050] Figure 5 yes Figure 4 Flow chart of step S404 in FIG.

[0051] Figure 6 yes Figure 4 Flow chart of step S402 in FIG.

[0052] Figure 7 yes Figure 4 Flow chart of step S404 in FIG.

[0053] Figure 8 It is a structural schematic diagram of a web page data collection device provided in an embodiment of the present application;

[0054] Fig. 9It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0056] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0057] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0058] First, some nouns involved in this application are analyzed:

[0059] Page View (PV): refers to the total number of times a page on a website or application is visited. Each time a page is loaded or refreshed, the page view count will increase, which is used to reflect the traffic of the website or application.

[0060] Unique Visitors (UV): refers to the number of unique users who visit a website or application. Unique Visitors are counted by distinguishing the user's unique identifier (such as user ID and cookie). Even if the same user visits the page multiple times, the number of unique visitors will only be recorded once.

[0061] In related technologies, behavioral information such as page views, bounce rates, and user dwell time are collected through web analysis tools or screen recording technology based on Web Real-Time Communication (WebRTC) to help understand user needs and optimize web design in a timely manner.

[0062] However, using web analytics tools has a limit on the data storage time, and the data cannot be stored permanently. In addition, web data analysts cannot directly access the original data, and can usually only view and analyze the data through predefined reports and dashboards of web analytics tools. This limitation means that web data analysts cannot obtain more fine-grained interaction data. In addition, although web analytics tools provide data export functions, the format and content of the exported data are limited, and can often only be exported in formats such as CSV or Excel. The dimensions and indicators of the exported data are preset and cannot be customized according to business needs, which in turn limits the flexibility of controlling web-related data.

[0063] In another related screen recording technology based on Web Real-Time Communication (WebRTC), all operations in the page are recorded until the page is closed.

[0064] The above two related technologies will acquire and record all operations of independent web visitors on the web page, resulting in too much data sampling, too much storage resources occupied by web page data from the time the web page is opened to the time it is closed, and the data complexity is also high. Ultimately, a lot of time will be spent on analyzing behavioral data, which reduces the efficiency of web page optimization.

[0065] Based on this, the embodiments of the present application provide a web page data collection method and device, an electronic device and a storage medium, aiming to improve the efficiency of web page optimization.

[0066] The web page data collection method and device, electronic device and storage medium provided in the embodiments of the present application are specifically described through the following embodiments. First, the web page data collection method in the embodiments of the present application is described.

[0067] The web page data collection method provided in the embodiment of the present application relates to the technical field of web page data collection. The web page data collection method provided in the embodiment of the present application can be applied to a terminal, can also be applied to a server side, and can also be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or can be configured as a server cluster or a distributed system composed of multiple physical servers, and can also be configured as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the web page data collection method, etc., but is not limited to the above forms.

[0068] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0069] Figure 1 is an optional flow chart of the web page data collection method provided in the embodiment of the present application. Figure 1 The method may include but is not limited to steps S101 to S104.

[0070] Step S101, obtaining a target web page.

[0071] Step S102, in response to the access operation to the target webpage, obtaining the network quality status, data transmission volume and original webpage operation of the target webpage.

[0072] Step S103, determining a web page collection frequency according to at least one of a network quality status, a data transmission volume, and a web page sub-operation.

[0073] Step S104: collecting target web pages according to the web page collection frequency.

[0074] In the steps S101 to S104 shown in the embodiment of the present application, the scope and target of data collection can be clarified by first acquiring the target web page. Subsequently, in response to the access operation, the network quality status, data transmission volume, and the user's original web page operation are further acquired to ensure that the data collection process can adapt to the actual network conditions and visitor operations, and then dynamically adjust the collection strategy according to the current network conditions and operation requirements. Finally, the web page data is collected in real time according to the web page collection frequency determined in the above steps. Compared with the solution of collecting all data in full in the prior art, the method of this embodiment can adaptively adjust the web page collection frequency according to different environmental conditions during the collection process, effectively reducing the amount of collected data, thereby avoiding the waste of storage resources and reducing data complexity, thereby reducing the time and complexity required for subsequent analysis of collected data and formulation of web page optimization strategies, and ultimately improving the efficiency of web page optimization.

[0075] In step S101 of some embodiments, the target web page refers to the web page to be accessed determined by user input or other triggering conditions. The process of obtaining the target web page includes parsing the Uniform Resource Locator (URL), sending an HTTP request and obtaining a response. During the network request process, the system may check the browser cache to optimize the loading time. After successfully obtaining the web page, the system will load the HTML document of the web page and its related resources, such as Cascading Style Sheets (CSS), JavaScript and image files, into the memory to provide data support for subsequent steps.

[0076] In step S102 of some embodiments, the initiator of the access operation to the target web page is the visitor user. In other words, when the visitor opens the target web page, the real-time network quality status, data transmission volume and original web page operation are obtained. The original web page operation includes web page sub-operations. Specifically, the original web page operation is the operation that occurs to the target web page by the visitor, including web page sub-operations and expected operations pre-set for the web page, such as: the web page can be opened normally and smoothly, the pictures of the web page can be displayed normally, and the various interactive operations of the visitor can be responded to in a timely manner. The web page sub-operation refers to the data of various interactive operations performed by the visitor in the current interface, such as mouse movement, mouse clicks on a specific area in the target web page, mouse scrolling, keyboard input of text, and changing the size of the window, etc., but is not limited to this. In some embodiments, the network status event of the browser can be detected to obtain the current network quality status to determine whether the network is normally connected or in a weak network state.

[0077] In other embodiments, the network status can be obtained according to JavaScript properties. Taking navigator.online as an example, when the Boolean value returned by this property is true, it indicates that the current network status is online. When the Boolean value returned by this property is false, it indicates that the current network status is offline. Furthermore, when the network status is online, navigator.connection.effectiveType can be used to obtain the network type, bandwidth, and round trip time (RTT) of the current network connection to further evaluate the performance quality of the currently connected network.

[0078] In some embodiments, the browser's performance API (such as Performance Navigation and Performance Resource Timing) can be used to track the amount of data transferred during page loading, including the number of bytes requested, response time, etc., so as to evaluate the response speed of the page.

[0079] In some embodiments, the original web page operation can be obtained by using DOM recording technology, which is a method of recording operations by monitoring and capturing user interaction behaviors on a web page. DOM recording mainly relies on JavaScript event detection mechanism, which can capture a variety of user interaction events, such as clicks, inputs, and scrolls. While capturing these operations, the technology also records a snapshot of the state of the current web page, including information such as the attributes and styles of DOM elements. This information will be serialized into a storable format, such as JSON and operation log files, to facilitate subsequent data processing and analysis.

[0080] In step S103 of some embodiments, it should be noted that the data finally collected by the webpage data collection method provided in the embodiments of the present application includes the video data of the current interface and the data related to the original webpage operation. Therefore, the webpage collection frequency includes the operation page collection frequency and the operation data collection frequency, which correspond to two types of data respectively.

[0081] See also Figure 2 In some embodiments, step S103 may include but is not limited to steps S201 to S203:

[0082] Step S201: determining an operation page collection frequency according to at least one of a network quality status and a data transmission volume.

[0083] Step S202, determining the frequency of collecting operation data according to the web page sub-operation.

[0084] Step S203, determining the web page collection frequency according to the operation page collection frequency and the operation data collection frequency.

[0085] In step S201 of some embodiments, the operation page acquisition frequency refers to the acquisition frequency of recording the target web page. The network quality status includes a normal connection state and a weak network state. Specifically, the type of the currently connected network can be obtained to determine whether it is in a weak network state. For example, if the type of the currently connected network is 5G, the network quality state is determined to be a normal connection state. If the type of the currently connected network is 2G, 3G or 4G, the network quality state is determined to be a weak network state.

[0086] Furthermore, when the network quality status is a normal connection status, the operation page acquisition frequency is increased to make the collected video data smoother. When the network quality status is a weak network status, the operation page acquisition frequency is reduced to reduce the amount of collected video data. It should be noted that the specific value of the operation page acquisition frequency can be preset according to different network quality types, and the preset database or data table is queried according to the real-time network quality status to obtain the operation page acquisition frequency. This embodiment does not strictly limit the specific value of the operation page acquisition frequency.

[0087] In some embodiments, when the data transmission volume is greater than or equal to a preset transmission volume threshold, it is determined that the total amount of data transmitted by the target page through the network is too large, and the operation page acquisition frequency is reduced to reduce the amount of video data. When the data transmission volume is less than the preset transmission volume threshold, it is determined that the total amount of data transmitted by the target page through the network is moderate, and the operation page acquisition frequency setting is increased to collect more detailed and smoother video data. Furthermore, a mapping table can be preset to represent the mapping relationship between the data transmission volume interval and the operation page acquisition frequency, and the selected value of the operation page acquisition frequency is determined according to the interval into which the data transmission volume falls.

[0088] In some embodiments, a mapping table may be pre-set to comprehensively consider the impact of data transmission volume and network quality status on the frequency of operation page acquisition, as shown in Table 1:

[0089] Network quality status Data transfer volume Operation page collection frequency Normal connection status Too large 30fps Normal connection status Moderate 60pfs Weak network connection status Too large 15fps Weak network connection status Moderate 30fps

[0090] Table 1

[0091] In step S202 of some embodiments, see Figure 3 In some embodiments, step S202 may include but is not limited to steps S301 to S304:

[0092] Step S301, obtaining the operation type of the web page sub-operation.

[0093] Step S302: query the preset collection frequency table according to the operation type to obtain the operation data collection frequency.

[0094] In step S301 of some embodiments, the operation type of the web page sub-operation may be mouse movement, mouse scrolling, mouse clicking, text input, window position change, window size change, etc., but is not limited thereto.

[0095] In step S302 of some embodiments, the preset collection frequency table is used to characterize the mapping relationship between the operation type and the collection frequency. The operation data collection frequency is the frequency of collecting data related to the visitor's interactive operation.

[0096] Exemplarily, if the type of web page sub-operation belongs to events such as text input, high-frequency click operation on the target web page, window position change, and window size change, the frequency of operation data collection can be reduced, or the final value can be directly collected. If the type of web page sub-operation belongs to some events that have little effect on the restoration of visitor behavior, such as focus events: visitors click or otherwise select input fields (such as text boxes, drop-down menus, buttons, etc.). And loss of focus (blur) events: visitors click or switch from the input field to other elements, causing the input field to lose its active state. This is usually used to verify input data, hide prompt information, or restore the default style of elements. When it is determined that the visitor is performing these operations, the frequency of operation data collection can be directly set to zero, that is, no collection is performed. Different operation types can be queried according to the preset collection frequency table to obtain the expected collection frequency.

[0097] In steps S301 to S302 shown in the embodiment of the present application, by obtaining the operation type of the web page sub-operation, the system can identify and record the user's specific interactive behavior in detail, and then adjust the frequency of operation data collection in real time according to the operation type. For key visitor interactions, the frequency of operation data collection will be increased to ensure that detailed data is obtained, while for minor interactive operations, the collection frequency can be appropriately reduced to avoid data redundancy. This flexibility not only improves the relevance and effectiveness of the data, but also reduces the consumption of storage and processing resources.

[0098] In other embodiments, the type of web page sub-operation may also affect the operation page acquisition frequency. For example, assuming that the initial operation page acquisition frequency is to collect the pixel coordinate data of the mouse 60 times per second, when the type of web page sub-operation belongs to mouse movement and mouse scrolling, the operation page acquisition frequency can be appropriately reduced and set to collect the pixel coordinate data of the mouse 10 times per second.

[0099] In step S203 of some embodiments, the web page collection frequency includes the collection frequency of video data and the collection frequency of visitor interaction operation data, so as to ensure that the video data and visitor interaction operation data can be collected separately according to the operation page collection frequency and the operation data collection frequency according to the actual network environment, and the amount of collected data can be adjusted in real time.

[0100] Steps S201 to S203 shown in the embodiment of the present application can dynamically adjust the data collection strategy by determining the collection frequency of the operation page according to the network quality status and data transmission volume, ensuring that more visitor behavior data is recorded when the network conditions are good, and reducing data collection when the network conditions are poor, thereby reducing the network burden and improving the visitor user experience. The collection frequency of the operation data is then determined according to the operation type of the web page sub-operation, and the appropriate collection frequency can be set for different visitor interaction behaviors. The method of this embodiment improves the quality and effectiveness of the data and also reduces the consumption of storage and processing resources.

[0101] In step S104 of some embodiments, video data of the target web page is collected according to the operation page collection frequency, and visitor interaction operation data is collected according to the operation data collection frequency.

[0102] After step S104 in some embodiments, the functions of the prior art are all single. If you want to collect visitor PV, UV and other data, you need to access a set of solutions to implement it. If you want to collect user abnormal information, you need to access another set of monitoring solutions to implement it. The two functions are not integrated, so the abnormal situation of the page cannot be better restored.

[0103] Specifically, see Figure 4 The webpage data collection method provided in the embodiment of the present application may also include but is not limited to steps S401 to S403:

[0104] Step S401, performing webpage anomaly detection on the target webpage to obtain an anomaly timestamp.

[0105] Step S402: acquiring original web page data obtained by collecting the target web page.

[0106] Step S403: annotate the original web page data according to the abnormal timestamp to obtain the target web page data.

[0107] Step S404, storing the target web page data in a preset database.

[0108] In step S401 of some embodiments, web page anomaly detection includes, but is not limited to, high-frequency operations of the target page by the visitor, the inability to open the web page, a white screen of the web page, no response to visitor clicks, page freezes and crashes, and no display of pictures. The abnormal timestamp refers to the time when the above abnormal event occurs. In addition, the specific information of the abnormal event, the browser version, operating system and stack used by the visitor can also be obtained. It is understandable that the occurrence of web page anomalies may be related to information such as the browser version, operating system and device model. Providing this information can better restore the scene where the error occurs, locate the problem and solve the problem.

[0109] In step S402 of some embodiments, the original web page data includes video screen data and relevant data of the original web page operation collected from the opening to the closing of the target page, including whether the web page is opened, displayed or jumped normally, and the visitor interaction operations mentioned in the above embodiments.

[0110] In some embodiments, the video screen data of the target web page can be collected through rrweb screen recording technology.

[0111] In step S403 of some embodiments, after an abnormal event is detected and a corresponding abnormal timestamp is generated, these timestamps will serve as key reference points to help data analysts quickly locate the moment when the event occurred. For example, if a visitor suddenly encounters a page crash while watching a video, the exact time when the crash occurred will be recorded. Subsequently, in the acquisition video of the original web page data, this time point will be marked as a jump point, which can be displayed on the progress bar of the video. Analysts can jump directly to the time points where abnormal events are marked and observe the user's behavior, operations, and page status before and after the abnormality occurs. In some embodiments, different types of marks can be set to different colors so that subsequent analysis can more intuitively determine the abnormal events that need to be traced back.

[0112] The target web page data includes video data with error traceback marks and data related to the original web page operation.

[0113] In step S404 of some embodiments, please refer to Figure 5 Step S404 includes but is not limited to steps S501 to S504:

[0114] Step S501: Acquire a first network connection status.

[0115] Step S502: if the first network connection state is network connection, the target web page data is stored in a preset database.

[0116] Step S503: if the first network connection status is that the network is not connected, the target web page data is stored in the local cache, the current network connection status is obtained, and the second network connection status is obtained.

[0117] Step S504: extract the target web page data from the local cache according to the second network connection state, obtain the intermediate web page data, and store the intermediate web page data in a preset database.

[0118] In step S501 of some embodiments, the first network connection state is used to indicate that the network is connected or not connected.

[0119] In some embodiments, the network status event of the browser can be detected to obtain the current network connection status and determine whether the network is online or offline. When the network status changes, relevant events are triggered. For example, when the network changes from offline to online, window.addEventListener('online') is triggered. When the network changes from online to offline, window.addEventListener('offline') is triggered.

[0120] In step S502 of some embodiments, the target web page data is stored in a database in a structured form to facilitate subsequent query and analysis. The target web page data may be packaged, including encryption, fragmented storage, and data filtering.

[0121] In step S503 of some embodiments, the second network connection status is used to indicate that the network connection is restored or the network connection is not restored.

[0122] In step S504 of some embodiments, the second network connection state is a network unrecovered state, and the target web page data continues to be stored in the local cache.

[0123] Steps S501 to S504 shown in the embodiment of the present application store the target web page data in the local cache in the disconnected state, thereby avoiding the risk of data loss, and extracting the intermediate web page data from the local cache according to the network recovery situation, and storing it in the preset database. This ensures that all collected data can eventually be effectively stored and used, and even in the case of network fluctuations, the integrity of the data is also guaranteed, optimizing the process of data collection and storage, and greatly improving the stability of the system and user experience.

[0124] Steps S401 to S404 shown in the embodiment of the present application detect anomalies on the target web page, timely identify and record abnormal events in user interactions, generate abnormal timestamps, annotate the original data according to the abnormal timestamps, and create backtracking points in the progress bar of the video data, allowing analysts to quickly locate the specific moment when the abnormal event occurred, greatly improving the efficiency and accuracy of abnormal analysis. The method of this embodiment not only effectively reduces the complexity and time cost of data analysis, but also enhances the ability to monitor and understand abnormal user behavior, thereby providing more powerful support for web page optimization.

[0125] In some embodiments, the collected data needs to be desensitized to ensure data security. Figure 6 Step S402 may include but is not limited to steps S601 to S606:

[0126] Step S601, acquiring initial web page data obtained by collecting a target web page, wherein the initial web page data includes web page image data and operation log data.

[0127] Step S602, performing sensitive area detection on the web page image data to obtain the sensitive area.

[0128] Step S603, performing image masking on the sensitive area to obtain the target web page image.

[0129] Step S604, obtaining the confidentiality level of the operation log data.

[0130] Step S605: encrypt the operation log data according to the confidentiality level to obtain encrypted log data.

[0131] Step S606, obtaining original web page data according to the target web page image and the encrypted log data.

[0132] In step S601 of some embodiments, the web page image data is an image of the target web page during the visitor's operation on the target web page, and may also be video data. The operation log data records the visitor's interactive operations on the target web page and the operation of the web page.

[0133] In step S602 of some embodiments, the sensitive area refers to an area containing user personal information, sensitive data or other content that needs to be protected, such as an input box, a password field or personal profile information. Sensitive areas can be identified by an object vision detection algorithm to ensure that user privacy can be properly protected in subsequent processing.

[0134] In step S603 of some embodiments, masking or blurring is applied to the sensitive area to prevent leakage of sensitive information during storage or analysis.

[0135] In step S604 of some embodiments, the confidentiality level may include a low confidentiality level and a high confidentiality level, and the operation log data involving user personal information, sensitive data or other data that needs to be protected is set to have a high confidentiality level. Exemplarily, the operation log data obtained is [user A, enter 12345 in the ID card number information input box], and the confidentiality level of the operation log data can be determined to be a high confidentiality level.

[0136] In step S605 of some embodiments, the operation log data may be hashed using the Advanced Encryption Standard, the Data Encryption Standard, triple DES, asymmetric encryption, or a hash function.

[0137] In step S606 of some embodiments, original web page data is generated based on the desensitized target web page image and encrypted log data. The original web page data at this time is protected web page content, including processed images and securely encrypted operation logs, ensuring that sensitive information is not leaked during analysis and use.

[0138] Steps S601 to S606 shown in the embodiment of the present application, by effectively combining data collection, sensitive information processing and security strategies, not only improve the security and privacy protection of web page data, but also ensure the validity and reliability of the data, providing a solid guarantee for subsequent analysis and web page optimization.

[0139] See also Figure 7 In some embodiments, step S404 may include but is not limited to steps S701 to S704:

[0140] Step S701, obtaining the data storage capacity of the target web page data.

[0141] Step S702: if the data storage capacity is greater than or equal to the preset storage capacity, the target web page data is sliced ​​to obtain sliced ​​web page data.

[0142] Step S703, compress the fragmented web page data to obtain compressed web page data.

[0143] Step S704: storing the compressed web page data in a preset database.

[0144] In step S701 of some embodiments, the data storage capacity of the target web page data can be obtained in a variety of ways. For example, the browser provides some APIs, such as navigator.storage.estimate(), which can help developers obtain the current storage usage and available storage capacity. This method returns a Promise, which will provide an object containing the used storage and total storage after parsing. Another way is to query the back-end database, use SQL statements to query the number of rows and field sizes of the table, or use the built-in functions of the database management system to obtain the total storage capacity of the table. Finally, in the case of using Web storage (such as LocalStorage or IndexedDB) to store web page data, the system can query the usage of these storages to obtain the specific amount of data stored.

[0145] In step S702 of some embodiments, if the data storage capacity is less than the preset storage capacity, the target web page data is directly stored in the database. If it is greater than or equal to the preset storage capacity, it is determined that the data volume of the target web page data is too large, and it is cut and segmented, and the segmented sub-data is the sliced ​​web page data. Exemplarily, 10 seconds is set as the time interval, and the target web page data collected within 1 minute is sliced ​​to obtain 6 pieces of sliced ​​web page data.

[0146] In step S703 of some embodiments, the fragmented web page data can be compressed according to Gzip compression, Brotli compression, Zlib compression and existing image compression algorithm libraries (such as pako, lz-string, compression.js, etc.), and the compression result is compressed web page data.

[0147] It should be noted that even if the target webpage data does not need to be sliced, the target webpage data can also be compressed before being stored in the database.

[0148] In step S704 of some embodiments, the preset database may be a relational database, such as MySQL, PostgreSQL, and Oracle, or a time series database, such as InfluxDB and TimescaleDB.

[0149] Steps S701 to S704 shown in the embodiment of the present application effectively improve the data transmission speed and reduce the demand for bandwidth by compressing the fragmented web page data, thereby greatly improving the storage and processing efficiency while ensuring data integrity, laying the foundation for efficient web page data management.

[0150] See also Figure 8 The embodiment of the present application also provides a web page data collection device, which can implement the above web page data collection method, and the device includes:

[0151] The first acquisition module is used to acquire the target web page.

[0152] The second acquisition module is used to acquire the network quality status, data transmission volume and original web page operation of the target web page in response to the access operation to the target web page; wherein the original web page operation includes web page sub-operation.

[0153] The acquisition frequency module is used to determine the web page acquisition frequency according to at least one of the network quality status, the data transmission volume and the web page sub-operation.

[0154] The data collection module is used to collect the target web page according to the web page collection frequency.

[0155] The specific implementation of the web page data collection device is substantially the same as the specific implementation of the web page data collection method described above, and will not be described in detail herein.

[0156] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above web page data collection method when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0157] See also Fig. 9 , Fig. 9 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0158] The processor 901 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0159] The memory 902 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 902, and the processor 901 calls and executes the webpage data collection method of the embodiment of this application;

[0160] Input / output interface 903, used to implement information input and output;

[0161] Communication interface 904, used to realize communication interaction between the device and other devices, which can be realized by wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);

[0162] A bus 905 that transmits information between various components of the device (e.g., the processor 901, the memory 902, the input / output interface 903, and the communication interface 904);

[0163] The processor 901 , the memory 902 , the input / output interface 903 and the communication interface 904 are connected to each other in communication within the device via a bus 905 .

[0164] The embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned web page data collection method is implemented.

[0165] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0166] The web page data collection method, web page data collection device, electronic device and storage medium provided in the embodiment of the present application can first clarify the scope and target of data collection by acquiring the target web page. Then, in response to the access operation, the network quality status, data transmission volume and the user's original web page operation are further obtained to ensure that the data collection process can adapt to the actual network conditions and visitor operations, and then dynamically adjust the collection strategy according to the current network conditions and operation requirements. Finally, the web page data is collected in real time according to the web page collection frequency determined in the above steps. Compared with the solution of collecting all data in full in the prior art, the method of this embodiment can adaptively adjust the web page collection frequency according to different environmental conditions during the collection process, effectively reducing the amount of collected data, thereby avoiding the waste of storage resources and reducing data complexity, thereby reducing the time and complexity required for subsequent analysis of collected data and formulation of web page optimization strategies, and ultimately improving the efficiency of web page optimization.

[0167] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0168] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0169] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0170] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0171] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0172] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0173] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0174] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0175] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0176] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes multiple instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, and other media that can store programs.

[0177] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A web page data collection method, characterized in that: The method comprises: Get the target web page; In response to the access operation to the target webpage, the network quality status, data transmission volume and original webpage operation of the target webpage are obtained; wherein the original webpage operation includes webpage sub-operations; determining a web page acquisition frequency according to at least one of the network quality status, the data transmission volume, and the web page sub-operation; The target web page is collected according to the web page collection frequency.

2. The webpage data collection method according to claim 1, characterized in that: After collecting the target web page according to the web page collection frequency, the method further includes: Performing webpage anomaly detection on the target webpage to obtain an anomaly timestamp; Acquiring original web page data obtained by collecting the target web page; Marking the original web page data according to the abnormal timestamp to obtain target web page data; The target web page data is stored in a preset database.

3. The webpage data collection method according to claim 2, characterized in that: The obtaining of original webpage data obtained by collecting the target webpage includes: Acquire initial web page data obtained by collecting the target web page; wherein the initial web page data includes web page image data and operation log data; Performing sensitive area detection on the webpage image data to obtain sensitive areas; Performing image masking on the sensitive area to obtain a target web page image; Obtaining the confidentiality level of the operation log data; Encrypting the operation log data according to the confidentiality level to obtain encrypted log data; The original web page data is obtained according to the target web page image and the encrypted log data.

4. The webpage data collection method according to claim 2, characterized in that: The storing the target webpage data into a preset database comprises: Acquire the data storage capacity of the target webpage data; If the data storage capacity is greater than or equal to the preset storage capacity, slicing the target web page data to obtain sliced ​​web page data; Compressing the fragmented web page data to obtain compressed web page data; The compressed web page data is stored in the preset database.

5. The webpage data collection method according to claim 2, characterized in that: The storing the target webpage data into a preset database comprises: Acquire a first network connection status; the first network connection status is used to indicate that the network is connected or not connected; If the first network connection state is network connection, storing the target web page data in the preset database; If the first network connection state is that the network is not connected, the target web page data is stored in a local cache, the current network connection state is acquired, and a second network connection state is obtained; The target web page data is extracted from the local cache according to the second network connection state to obtain intermediate web page data, and the intermediate web page data is stored in the preset database.

6. The webpage data collection method according to claim 1, characterized in that: The webpage collection frequency includes the operation page collection frequency and the operation data collection frequency; The determining of the web page acquisition frequency according to at least one of the network quality status, the data transmission volume and the web page sub-operation includes: Determining the operation page collection frequency according to at least one of the network quality status and the data transmission volume; Determining the operation data collection frequency according to the webpage sub-operation; The web page collection frequency is determined according to the operation page collection frequency and the operation data collection frequency.

7. The webpage data collection method according to claim 6, characterized in that: The determining the operation data collection frequency according to the webpage sub-operation includes: Obtaining the operation type of the webpage sub-operation; The preset acquisition frequency table is queried according to the operation type to obtain the operation data acquisition frequency; wherein the preset acquisition frequency table is used to characterize the mapping relationship between the operation type and the acquisition frequency.

8. A web page data collection device, characterized in that: include: A first acquisition module, used to acquire a target web page; A second acquisition module, configured to acquire the network quality status, data transmission volume, and original web page operation of the target web page in response to the access operation to the target web page; wherein the original web page operation includes web page sub-operations; A collection frequency module, used to determine a web page collection frequency according to at least one of the network quality status, the data transmission volume and the web page sub-operation; The data collection module is used to collect the target web page according to the web page collection frequency.

9. An electronic device, characterized in that: The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the web page data collection method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the web page data collection method according to any one of claims 1 to 7 is implemented.

Images