Data acquisition method and device, computer equipment and storage medium

Abstract

The embodiment of the invention provides a data acquisition method and device, computer equipment and a storage medium, relates to the technical field of data acquisition, and is suitable for the fields of financial science and technology, digital medical treatment and the like. The method comprises the following steps: acquiring data acquisition configuration parameters of a target front-end webpage; wherein the data acquisition configuration parameters comprise acquisition webpage information and acquisition mode information; performing configuration processing on a preset original acquisition module according to the acquisition webpage information and the acquisition mode information to obtain a target acquisition module; wherein the target acquisition module comprises at least one target data acquisition unit; obtaining an original webpage hierarchy of the target front-end webpage and hierarchy setting information of the original webpage hierarchy; screening out a target webpage hierarchy from the original webpage hierarchy based on the hierarchy setting information; and performing data acquisition on the target webpage hierarchy through each target data acquisition unit to obtain target data. According to the embodiment of the invention, the data acquisition manpower can be reduced, and the data acquisition efficiency is improved.

Description

Technical Field

[0001] The present application relates to the field of data collection technology, and is applicable to the fields of financial technology, digital medicine, etc., and in particular to a data collection method and device, computer equipment and storage medium. Background Art

[0002] The performance and user experience of the front-end web page affect the frequency of user use, so it is necessary to regularly collect data on the front-end web page for analysis. For example, on an insurance service web page, it is necessary to regularly collect user click events on the insurance service web page, and collect data related to click events when users enter the insurance service web page, so as to analyze the performance and user experience of the insurance service web page through the collected data.

[0003] In the related art, data collection of front-end web pages is mainly achieved by embedding codes, which monitor the data related to user click behavior through codes. However, the embedding code method requires setting corresponding codes for each component's embedding point, which consumes a lot of manpower and has low data collection efficiency. Therefore, how to reduce the manpower of data collection and improve the efficiency of data collection has become a technical problem that needs to be solved urgently. Summary of the invention

[0004] The main purpose of the embodiments of the present application is to propose a data collection method and apparatus, a computer device and a storage medium, aiming to reduce the manpower of data collection and improve the efficiency of data collection.

[0005] To achieve the above object, a first aspect of an embodiment of the present application provides a data collection method, the method comprising:

[0006] Acquire data collection configuration parameters of the target front-end webpage; wherein the data collection configuration parameters include: collection webpage information and collection mode information;

[0007] According to the collected web page information and the collected mode information, a preset original collection module is configured and processed to obtain a target collection module; wherein the target collection module includes at least one target data collection unit;

[0008] Acquire the original webpage level of the target front-end webpage and level setting information of the original webpage level;

[0009] Filtering a target web page hierarchy from the original web page hierarchy based on the hierarchy setting information;

[0010] Each of the target data collection units collects data at the target web page level to obtain target data.

[0011] In some embodiments, the collected web page information includes: web page interface information, collection level information and collection attribute information, and the original collection module includes at least one original data collection unit;

[0012] The configuring and processing the preset original acquisition module according to the acquired web page information and the acquisition mode information to obtain the target acquisition module includes:

[0013] According to the webpage interface information and the collection level information, each of the raw data collection units is configured with a collection address to obtain a preliminary data collection unit;

[0014] According to the collection attribute information, the preliminary data collection unit is configured with collection attributes to obtain a candidate data collection unit;

[0015] Performing collection mode configuration on each of the candidate data collection units according to the collection mode information and the collection level information to obtain the target data collection unit;

[0016] The target data acquisition units are spliced ​​to obtain the target acquisition module.

[0017] In some embodiments, configuring the collection mode of each candidate data collection unit according to the collection mode information and the collection level information to obtain the target data collection unit includes:

[0018] Acquire hierarchical function information of each level in the target front-end webpage according to the collected hierarchical information;

[0019] Filtering target mode information from the acquisition mode information according to the hierarchical function information;

[0020] The target data collection unit is obtained by configuring the collection mode of each candidate data collection unit according to the target mode information.

[0021] In some embodiments, after collecting data from the target web page level by each of the target data collection units to obtain target data, the method further includes:

[0022] Performing a reporting trigger evaluation on the target data according to a preset reporting trigger rule to obtain reporting trigger evaluation information; wherein the reporting trigger evaluation information indicates whether the target data is reported or not reported;

[0023] If the reporting trigger evaluation information indicates that the target data is to be reported, the target data is reported to the target platform.

[0024] In some embodiments, the reporting triggering rule includes: triggering event information and triggering node information;

[0025] The step of performing a reporting trigger evaluation on the target data according to the reporting trigger rule to obtain reporting trigger evaluation information includes:

[0026] Analyze the target data to obtain data analysis information;

[0027] Extracting target event information from the target data according to the data parsing information and the trigger node information;

[0028] Compare the target event information with the trigger event information to obtain event comparison information;

[0029] The target data is subjected to a reporting trigger evaluation according to the event comparison information to obtain the reporting trigger evaluation information.

[0030] In some embodiments, if the reporting triggering evaluation information is characterized as reporting the target data, reporting the target data to the target platform includes:

[0031] If the reporting trigger evaluation information indicates that the target data is to be reported, acquiring collection unit information of the target data;

[0032] Encrypting the target data to obtain encrypted data;

[0033] Prioritize the encrypted data according to the acquisition unit information to obtain data priority information;

[0034] The encrypted data is reported to the target platform according to the data priority information.

[0035] In some embodiments, after collecting data from the target web page level by each of the target data collection units to obtain target data, the method further includes:

[0036] receiving configuration update requests;

[0037] Update the data collection configuration parameters according to the configuration update request to obtain updated collection configuration parameters;

[0038] According to the updated acquisition configuration parameters, the target data acquisition unit is configured and updated to obtain an updated data acquisition unit;

[0039] The update data collection unit collects data from the target front-end webpage to obtain update data.

[0040] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application provides a data acquisition device, the device comprising:

[0041] The parameter acquisition module is used to acquire the data acquisition configuration parameters of the target front-end webpage; wherein the data acquisition configuration parameters include: acquisition webpage information and acquisition mode information;

[0042] A configuration module, configured to configure a preset original acquisition module according to the acquisition webpage information and the acquisition mode information to obtain a target acquisition module; wherein the target acquisition module includes at least one target data acquisition unit;

[0043] An information acquisition module, used to acquire the original webpage level of the target front-end webpage and level setting information of the original webpage level;

[0044] A level screening module, used for screening out a target web page level from the original web page level based on the level setting information;

[0045] The data collection module is used to collect data from the target web page level through each of the target data collection units to obtain target data.

[0046] To achieve the above-mentioned purpose, the third aspect of an embodiment of the present application proposes a computer device, which includes a memory and a processor, the memory stores a computer program, and the processor implements the method described in the first aspect when executing the computer program.

[0047] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in the first aspect is implemented.

[0048] The data collection method and device, computer equipment and storage medium proposed in the present application configure the original collection module into a target collection module according to the collection web page information and collection mode information configured by the target front-end web page, and then select the target web page level that allows data collection from the original web page level of the target front-end web page. When data collection is required, data collection of the target web page level is completed through each target data collection unit in the target collection module, and there is no need to configure the code for the components of the target front-end web page one by one in advance, making the data collection operation easier. Therefore, the data collection operation only needs to complete the collection configuration of the collection module in advance, realizing personalized data collection, and the data collection operation is simple, manpower is saved, and the efficiency of data collection is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 is a flow chart of a data collection method provided in an embodiment of the present application;

[0050] Figure 2It is a schematic diagram of the interface of the data collection platform in the data collection method provided in the embodiment of the present application;

[0051] Figure 3 yes Figure 1 Flow chart of step S102 in FIG.

[0052] Figure 4 yes Figure 3 Flow chart of step S303 in FIG.

[0053] Figure 5 is a flow chart of a data collection method provided by another embodiment of the present application;

[0054] Figure 6 yes Figure 5 Flow chart of step S501 in FIG.

[0055] Figure 7 yes Figure 5 Flow chart of step S502 in FIG.

[0056] Figure 8 is a flow chart of a data collection method provided by another embodiment of the present application;

[0057] Fig. 9 is an example flow chart of a data collection method provided in an embodiment of the present application;

[0058] Fig.10 It is a structural schematic diagram of a data acquisition device provided in an embodiment of the present application;

[0059] Fig.11 It is a schematic diagram of the hardware structure of the computer device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0060] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0061] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0062] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0063] First, some nouns involved in this application are analyzed:

[0064] Artificial intelligence (AI) is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence. AI is a branch of computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine that can respond in a similar way to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing and expert systems. AI can simulate the information process of human consciousness and thinking. AI is also a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0065] Data tracking: It is a commonly used data collection method. Data product managers, data operations and data analysts implant relevant codes into the pages and locations corresponding to each event of user behavior in the application based on business needs or product needs, and report statistical data through collection tools so that relevant personnel can track user behavior and application usage, promote product optimization or guide operations. Including visits, visitors, stay time, page views and bounce rate. Such information collection can be roughly divided into two types: page statistics (track this virtual page view) and statistical operation behavior (track this button by an event).

[0066] No tracking: Record the user's behavior on the front-end page indiscriminately. No tracking does not mean that you don't need to add code, but that developers don't need to add additional code. Whether it is a tracking method or a non-tracking method, basic code must be added. No tracking directly monitors all user behaviors on the page, so even if you don't need to monitor a certain part, the user behavior data and corresponding information of this part will still be recorded.

[0067] HTML elements: are the basic building blocks of HTML pages, consisting of a start tag, content, and an end tag.

[0068] Navigator: doNotTrack: is a browser property that indicates whether the user wants the website to track their browsing behavior. Three possible values ​​are returned: "yes": the user explicitly expressed that they do not want to be tracked. "no": the user explicitly expressed that they want to be tracked. "undefined" or null: the user did not express an explicit preference, or the browser does not support this property.

[0069] Persistence layer: responsible for data persistence, including the data layer, i.e., the database, and the data access layer. The database is the carrier for data persistence, and the data access layer is the interface for interaction between the business layer and the persistence layer. The business layer needs to persist data in the database through the data access layer. In layman's terms, the persistence layer interacts with the database and performs deletion, modification, and query on the database table.

[0070] In order to understand the user's usage habits and determine the shortcomings of the product, it is necessary to collect statistical data of the front-end web page through a collection tool so that relevant personnel can track user behavior and application usage. In the related technology, the main method is to collect statistical data of the front-end web page by using data embedding. Among them, there are currently three main types of data embedding, one of which is code embedding, that is, embedding in the form of embedded code. For example, if you need to monitor the user's click event, insert a piece of code when the user clicks, and obtain the behavioral data collection after the user clicks through the code. However, the use of code embedding requires a lot of manpower, and the corresponding code needs to be set for each component embedding. Another embedding method is visual embedding, which replaces code embedding by visual means, analyzes the business code and embedding code, and customizes the embedding in the business code, but still needs to set the code, which also requires a lot of manpower. In addition, the third is the non-embedding method, which captures all events on all DOM structures and parses the captured events regularly, but all event collection and parsing will increase server pressure.

[0071] Based on this, the embodiment of the present application provides a data collection method and device, a computer device and a storage medium, by setting a collection module, and configuring the collection module in advance according to the collection web page information and collection mode information in the data collection configuration parameters, and then determining the target web page level for data collection, so as to collect data for the target web page level through the data collection unit in the collection module. Therefore, not only can personalized data collection be achieved, but each data collection only needs to configure the collection module in advance, simplifying the operation of data collection, saving manpower for data collection, and improving the efficiency of data collection.

[0072] The data collection method and apparatus, computer equipment and storage medium provided in the embodiments of the present application are specifically described through the following embodiments. First, the data collection method in the embodiments of the present application is described.

[0073] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0074] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0075] The data collection method provided in the embodiment of the present application relates to the field of data collection technology and is applicable to the fields of financial technology, digital medicine, etc. The data collection method provided in the embodiment of the present application can be applied to the terminal, can also be applied to the server side, and can also be software running in the terminal or the server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or it can be configured as a server cluster or distributed system composed of multiple physical servers, and can also be configured as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the data collection method, etc., but is not limited to the above forms.

[0076] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer computer devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0077] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0078] Figure 1 is an optional flow chart of the data collection method provided in the embodiment of the present application. Figure 1 The method may include but is not limited to steps S101 to S105.

[0079] Step S101, obtaining data collection configuration parameters of a target front-end webpage; wherein the data collection configuration parameters include: collection webpage information and collection mode information;

[0080] Step S102, configuring and processing the preset original acquisition module according to the acquired web page information and the acquisition mode information to obtain a target acquisition module; wherein the target acquisition module includes at least one target data acquisition unit;

[0081] Step S103, obtaining the original webpage hierarchy of the target front-end webpage and hierarchy setting information of the original webpage hierarchy;

[0082] Step S104, filtering out a target web page hierarchy from the original web page hierarchy based on the hierarchy setting information;

[0083] Step S105 , collecting data on the target web page level through each target data collection unit to obtain target data.

[0084] Steps S101 to S105 shown in the embodiment of the present application configure the original acquisition module according to the acquisition web page information and acquisition mode information configured in the target front-end web page to obtain the target acquisition module. Among them, the target acquisition module is composed of at least one target data acquisition unit, and then determines the target web page level in the target front-end web page where data can be collected, and obtains the target data by collecting data from the target web page level through each target data acquisition unit. Therefore, in the data collection process, it is only necessary to configure the acquisition module in advance, which not only realizes personalized data collection configuration, but also simplifies the data collection operation, without the need to set codes for each buried point, saving manpower in the data collection process and improving the efficiency of data collection.

[0085] In step S101 of some embodiments, the data collection method is applied to a data collection device, and the data collection device is represented in the form of a data collection platform. The data collection configuration parameters are parameters that are configured in advance for data collection on the target front-end web page, and the data collection configuration parameters can be configured on the data collection platform, or can be set in advance in other ways, but are not limited to this. Specifically, the data collection configuration parameters include: collection web page information and collection mode information. The collection web page information represents information related to the target front-end web page, which can be the address of the target front-end web page, the collection method, the collection data category, etc. The collection mode information is the collection mode of the target front-end web page, specifically the buried point function category.

[0086] For example, Figure 2 As shown, if the target front-end web page is an insurance business web page, a data collection configuration interface is set on the data collection platform, and collection web page configuration items and collection mode configuration items are set on the data collection configuration interface. Users can fill in the information that needs to be collected on the insurance business web page in the collection web page configuration items, including the address of the insurance business web page, collection method, data reporting method, etc. At the same time, the data collection mode can be selected at the collection mode configuration item, and the collection mode can be selected through the option box that pops up next to it to determine the collection mode information. After the data collection configuration parameters are set on the data collection configuration interface, a point-burying data packet will be generated, and the point-burying data packet needs to be sent to the server where the insurance business web page is located. By loading the point-burying data packet through the server, and introducing and registering it in the entry file of the server through the point-burying data packet, the data collection after all user click events on the insurance business web page can be automatically completed, realizing fully automated data collection operations.

[0087] In some embodiments, the collected web page information includes: web page interface information, collection level information and collection attribute information. It should be noted that the web page interface information is the interface information of the target front-end web page, the collection level information is the level information of the target front-end web page that needs to be collected, and the collection attribute information is the data attribute of the collected target front-end web page, and the data attribute is also called the data category, that is, determining the data category in the collected target front-end web page. Specifically, in this embodiment, the collected data categories include basic data categories, functional data categories, user behavior data categories, web page data categories, system data categories, etc. This embodiment does not make specific restrictions on data categories.

[0088] It should be noted that there are generally two forms of development architecture, one is C / S architecture, that is, client / server; the other is B / S architecture, that is, browser server. The target front-end web page is generally a B / S architecture, and in the B / S architecture, the system standard three-layer architecture includes: presentation layer, business layer and persistence layer, and each web page level has its own function. Among them, the presentation layer is used to receive requests from the user end and display the corresponding results to the user end, and the presentation layer also includes a display layer and a control layer. The control layer is responsible for request reception and the display layer is responsible for result display. The business layer is responsible for business logic processing, and the persistence layer is responsible for interacting with the database, that is, responsible for data persistence. Therefore, in order to achieve comprehensive data collection of the target front-end web page, the number of original data collection units set by the original collection module in this embodiment is greater than or equal to the web page level of the target front-end web page, so the original data collection units of this embodiment are set to more than 3, and most of the data of the target front-end web page can be captured.

[0089] See also Figure 3 In some embodiments, step S102 may include but is not limited to steps S301 to S304:

[0090] Step S301, configuring the collection address for each original data collection unit according to the web page interface information and the collection level information to obtain a preliminary data collection unit;

[0091] Step S302, configuring the collection attributes of the preliminary data collection unit according to the collection attribute information to obtain a candidate data collection unit;

[0092] Step S303, configuring the collection mode for each candidate data collection unit according to the collection mode information and the collection level information to obtain a target data collection unit;

[0093] Step S304, splicing the target data acquisition units to obtain a target acquisition module.

[0094] In step S301 of some embodiments, the original data collection unit is configured with a collection address according to the web page interface information and the collection level information, that is, the collection address on the original data collection unit is set to the address corresponding to the web page interface information and the collection level information. Specifically, the interface address information of the target front-end web page is first determined according to the web page interface information, and then the level address information is determined from the interface address information according to the collection level information, and the original data collection unit is configured with an address according to the level address information to obtain a preliminary data collection unit. Therefore, the preliminary data collection unit can collect data for the corresponding web page level in the target front-end web page.

[0095] In step S302 of some embodiments, as disclosed above, each preliminary data collection unit is responsible for data collection of the corresponding web page level in the target front-end web page to realize data collection of different data categories. Therefore, the data collection attribute information that needs to be configured for different preliminary data collection units is different. Specifically, firstly, the target attribute information is screened out from the collection attribute information according to the collection level information, that is, the data category that the preliminary data collection unit needs to collect is determined, and then the preliminary data collection unit is attribute configured according to the target attribute information to obtain the candidate data collection unit.

[0096] In step S303 of some embodiments, after determining the address and data category of data collection, it is necessary to determine in which mode the data collection is completed, and the data collection modes set for the candidate data collection units responsible for different web page levels are different. Therefore, based on the collection level information and the collection mode information, the collection mode is configured for each candidate data collection unit respectively to construct target data collection units with different collection modes. Therefore, each target data collection unit collects the data of the corresponding web page level in the target front-end web page with the configured collection mode and data category, so as to realize targeted data collection, without waiting for the data collection of one web page level before collecting the data of the next web page level, thereby improving the efficiency of data collection.

[0097] In step S304 of some embodiments, at least one target data collection unit is spliced, that is, at least one target data collection unit is combined into a target collection module according to the collection level information. Therefore, the target collection module formed by the combination of at least one target data collection unit can specifically complete the data collection of each web page level in the target front-end web page.

[0098] In steps S301 to S304 shown in this embodiment, the address of the data collection unit is configured according to the web page interface information and the collection level information, the collection attribute of the data collection unit is configured according to the collection attribute information, and finally the collection mode of the data collection unit is configured according to the collection level information and the collection mode information. The three configuration operations of the address, collection attribute and collection mode of the data collection unit can be completed. Then, the data collection unit can complete data collection in a targeted manner based on the configured address, collection attribute and collection mode to realize personalized data collection.

[0099] See also Figure 4 In some embodiments, step S303 may include but is not limited to steps S401 to S403:

[0100] Step S401, obtaining hierarchical function information of each level in the target front-end webpage according to the collected hierarchical information;

[0101] Step S402, filtering out target mode information from the collection mode information according to the hierarchical function information;

[0102] Step S403: configure the collection mode of each candidate data collection unit according to the target mode information to obtain the target data collection unit.

[0103] In step S401 of some embodiments, as disclosed above, different web page levels need to set different collection modes to adapt to data collection at different web page levels. Therefore, the hierarchical function information of each original web page level is first determined according to the collection level information, and the hierarchical function information is the function information of the original web page level, that is, the aforementioned request receiving function, result display function and data persistence, etc.

[0104] In step S402 of some embodiments, a mode mapping relationship table of hierarchical function information and collection mode information is set in advance, and the corresponding collection mode information is found in the mode mapping relationship table as the target mode information according to the hierarchical function information. It should be noted that the collection mode information includes: jump link burying mode, submit data burying mode and automatic burying mode. This embodiment does not impose specific restrictions on the collection mode information. Specifically, this embodiment completes the collection mode configuration of the candidate data collection unit based on the DomTracker plug-in, and other embodiments may use other plug-ins to complete the collection mode configuration.

[0105] In step S403 of some embodiments, the collection mode of the candidate data collection unit is configured according to the target mode information to obtain the target data collection unit. Therefore, the target data collection unit is a data collection unit having a collection mode corresponding to the target mode information. Specifically, if the collection mode of the target data collection unit is a jump link burying mode, data collection is performed in a link jump manner; if the collection mode of the target data collection unit is a submit data burying mode, data collection is performed in a data submission manner; if the collection mode of the target data collection unit is an automatic burying mode, data collection is performed automatically.

[0106] In steps S401 to S403 shown in this embodiment, the target mode information of the original web page level is determined according to the level function information of different original web page levels, and the collection mode of the candidate data collection unit is configured according to the target mode information, so that the collection mode configuration operation of the candidate data collection unit is more accurate, so that different web page levels can complete data collection in different collection modes.

[0107] In step S103 of some embodiments, the level setting information indicates whether the original web page level allows data collection, does not allow data collection, or data collection is not set. Specifically, by obtaining the level attribute parameters of the original web page level, and extracting the level setting information from the level attribute parameters, and the level setting information is also called tracking setting information, defined as Navigator: doNotTrack, and the tracking setting information is determined by the value in Navigator: doNotTrack. It should be noted that the value in Navigator: doNotTrack can be "1", "0" and "null", "1" indicates that the original web page level allows data tracking, "0" indicates that the original web page level does not allow data tracking, and "null" indicates that the original web page level has not set whether data tracking is allowed. As disclosed above, the persistence layer is used to persist data, so after determining the value in Navigator: doNotTrack to determine whether data tracking is allowed, it is also necessary to determine whether the persistence layer allows data tracking. When both allow data tracking, data collection is performed on the original web page level that allows data tracking of the target front-end web page, reducing the situation of arbitrary data leakage.

[0108] In step S104 of some embodiments, the hierarchy setting information is characterized as an original web page hierarchy that allows data tracking as a target web page hierarchy to determine a web page hierarchy at which a target data collection unit can perform data collection.

[0109] In step S105 of some embodiments, the page monitoring mark information is set in advance on the original web page level, so when the target data collection unit collects data on the target web page level, it first obtains the monitoring mark information of each target web page level, and then the target data collection unit determines the selected web page level from the target web page level according to the monitoring mark information and the collection level information, and then collects data on the selected web page level, so as to realize a one-to-one data collection method between the target data collection unit and the web page level. As disclosed above, the number of units of the target data collection unit is greater than or equal to the number of the original web page level, so the number of units of the target data collection unit is also greater than or equal to the number of the target web page level, therefore, most of the data of the target front-end web page can be collected through the target data collection unit, making the data collection more comprehensive.

[0110] Specifically, this embodiment sets four target data collection units, which are respectively a first function collection unit, a second function collection unit, a third function collection unit and a basic data collection unit; the first function collection unit is defined as "moudleLink", the second function collection unit is defined as "modulePage", and the third function collection unit is defined as "moduleSeesion"; the basic data collection unit is set at the outermost web page level of the target front-end web page, and is used to monitor the user's machine model, system, and home page information visited by the user, and to count PV and UV. When the user enters the home page of the target front-end web page, the user entry event is rendered, and the user's basic information is collected. In addition, the basic data collection unit also provides basic tool functions such as type judgment, traversal, inheritance, bind, etc.; json, base64, utf8 encoding and decoding capabilities; url parameter reading and writing functions; cookie, localStorage reading and writing capabilities; dom event binding capabilities; dom node query capabilities; info browser information acquisition capabilities. This embodiment does not make specific restrictions on the functions of the basic data collection unit.

[0111] See also Figure 5 In some embodiments, after step S105, the data collection method may also include but is not limited to steps S501 to S502:

[0112] Step S501, performing a reporting trigger evaluation on the target data according to a preset reporting trigger rule to obtain reporting trigger evaluation information; wherein the reporting trigger evaluation information indicates whether the target data is reported or not reported;

[0113] Step S502: If the reporting trigger evaluation information is characterized by reporting the target data, the target data is reported to the target platform.

[0114] In step S501 of some embodiments, the reporting trigger rule records the set event elements that can trigger the reporting of target data, and multiple candidate event elements are stored in the target data. Therefore, the target data is evaluated for reporting trigger according to the reporting trigger rule, that is, it is determined whether the candidate event elements in the target data match the trigger event elements set for triggering reporting. If so, the reporting trigger evaluation information is determined to be characterized as reporting the target data; if not, the reporting trigger evaluation information is determined to be characterized as not reporting the target data.

[0115] In step S502 of some embodiments, as disclosed above, if it is determined that the reporting trigger evaluation information is characterized as reporting the target data, the target data is reported to the target platform. It should be noted that the data collection configuration parameters also include: proxy address information, and the proxy address information represents the address of the reported data, the target platform is determined according to the proxy address information, and the target data is sent to the target platform according to the proxy address information. It should be noted that the target platform serves as an application platform for the target data.

[0116] For example, in this embodiment, if the target front-end webpage is an insurance service webpage, the target platform can be an insurance product recommendation platform, a webpage optimization platform, a user analysis platform, etc. If the target platform is an insurance product recommendation platform, after receiving the target data, the user behavior data is extracted, and the user's preference information for insurance products is analyzed based on the user behavior data, and insurance products are recommended based on the preference information. Therefore, by sending the collected target data to the application platform, the scope of data use can be increased.

[0117] In some embodiments, the reporting triggering rule includes: triggering event information and triggering node information; the triggering event information represents the event element that triggers data reporting, and the triggering node information represents the web page element node that triggers data reporting.

[0118] See also Figure 6 In some embodiments, step S501 may include but is not limited to steps S601 to S604:

[0119] Step S601, analyzing the target data to obtain data analysis information;

[0120] Step S602, extracting target event information from target data according to data analysis information and trigger node information;

[0121] Step S603, comparing the target event information with the trigger event information to obtain event comparison information;

[0122] Step S604: perform reporting trigger evaluation on the target data according to the event comparison information to obtain reporting trigger evaluation information.

[0123] In step S601 of some embodiments, a data analysis tool is used to analyze the target data to obtain data analysis information, and the data analysis tool includes any one of the following: Python tool, SAS tool, SPSS tool and Tableau tool, etc. This embodiment does not specifically limit the data analysis tool. It should be noted that the data analysis information records that the target data has a candidate event element, and the candidate event element is an html element.

[0124] In step S602 of some embodiments, as disclosed above, candidate event elements are first extracted from the target data according to the data parsing information, and then target event elements are extracted from the candidate event elements according to the triggering node information as target event information.

[0125] In steps S603 to S604 of some embodiments, the target event information and the trigger event information are compared, that is, the trigger event element and each target event element are compared to obtain event comparison information. If the event comparison information is characterized by the existence of a target event element identical to the trigger event element, the reporting of the trigger evaluation information is characterized as reporting the target data; if the event comparison information is characterized by the absence of a target event element identical to the trigger event element, the reporting of the trigger evaluation information is characterized as not reporting the target data. Therefore, by determining whether there is a target event element identical to the trigger event element in the target event element, the reporting trigger status of the target data can be determined. It should be noted that if the reporting of the trigger evaluation information is characterized by not reporting the target data, the collected target data will be cleaned up regularly to reduce the space occupied by the local database.

[0126] In steps S601 to S604 shown in this embodiment, data parsing information is determined by parsing the target data, target event information is extracted from the target data based on the data parsing information and the trigger node information, and the reporting trigger status of the target data is determined based on the comparison between the triggering event information and the target event information, making the reporting trigger evaluation of the target data more accurate and simple.

[0127] See also Figure 7 In some embodiments, step S502 includes but is not limited to steps S701 to S704:

[0128] Step S701, if the reporting trigger evaluation information is characterized by reporting the target data, acquiring the collection unit information of the target data;

[0129] Step S702, encrypting the target data to obtain encrypted data;

[0130] Step S703, setting the priority of the encrypted data according to the acquisition unit information to obtain data priority information;

[0131] Step S704: Report the encrypted data to the target platform according to the data priority information.

[0132] In step S701 of some embodiments, the collection unit information is information of a target data collection unit that collects target data. It should be noted that different web page levels have different data reporting priorities, so different target data collection units have different reporting priorities, and it is necessary to determine the collection unit information of the target data in advance, that is, determine the reporting priority of the target data.

[0133] In step S702 of some embodiments, in order to improve the security of target data transmission, the target data needs to be encrypted to obtain encrypted data. It should be noted that the target data is encrypted using a preset encryption algorithm, and the encryption of the target data is completed by OpenSSL or crypto module. In this embodiment, the encryption algorithm can be an md5 encryption algorithm or a salted encryption algorithm, and this embodiment does not specifically limit the encryption algorithm, and the encryption algorithm can be set according to demand.

[0134] In step S703 of some embodiments, candidate priority information corresponding to each acquisition unit information is set in advance, so the target priority information is screened out from the candidate priority information according to the acquisition unit information, and then the data priority information of the encrypted data is set according to the target priority information. For example, in this embodiment, the web page hierarchy relationship is represented as moduleLink>modulePage>moduleSeesion, then the target data acquisition unit for collecting the corresponding web page hierarchy is also set according to the web page hierarchy relationship, so the priority of the target data collected at the corresponding target web page hierarchy is set according to the web page hierarchy relationship.

[0135] In step S704 of some embodiments, the data priority information represents the priority of target data reporting, so reporting the target data to the target platform one by one according to the data priority information can allow important target data to be reported first, so that the target platform can give priority to the application of important target data.

[0136] It should be noted that the target data mainly includes: the ID information of the user terminal, the user agent address, the reporting timestamp of the target data, the URL of the web page currently browsed by the user, the URL of the web page previously browsed by the user, the reporting event category, the triggering event element, custom data, etc. This embodiment does not limit the specific content of the target data.

[0137] In steps S701 to S704 shown in the present embodiment, if the reporting trigger evaluation information indicates that the target data is to be reported, the target data is encrypted to obtain encrypted data, and then the data priority information of the encrypted data is set according to the hierarchical information of the encrypted data, so that the encrypted data is reported to the target platform according to the data priority information, which can not only make the collected data more securely reported, but also enable important data to be reported first, thereby reducing the omission of important data.

[0138] See also Figure 8 In some embodiments, after step S105, the data collection method may also include but is not limited to steps S801 to S804:

[0139] Step S801, receiving a configuration update request;

[0140] Step S802, updating the data collection configuration parameters according to the configuration update request to obtain updated collection configuration parameters;

[0141] Step S803, updating the configuration of the target data acquisition unit according to the updated acquisition configuration parameters to obtain an updated data acquisition unit;

[0142] Step S804, collecting data from the target front-end webpage through the update data collection unit to obtain update data.

[0143] In some embodiments, in step S801 to step S802, the user modifies the data collection configuration parameters on the data collection platform to form a configuration update request. It should be noted that the updated collection configuration parameters are the configuration parameters after the data collection configuration parameters are updated, and the updated collection configuration parameters are different from the data collection configuration parameters.

[0144] In step S803 of some embodiments, updating the collection configuration parameters includes any one of the following: updating web page information and updating mode information, and the target data collection unit can be updated by configuring the web page and / or collection mode according to the updated web page information and / or the updated mode information. Therefore, when the data collection of the target front-end web page needs to be changed, the user only needs to update the collection configuration parameters on the data collection platform, without rebuilding the code and loading it to the server where the target front-end web page is located, which simplifies the update operation of data collection and makes data collection easier.

[0145] In step S804 of some embodiments, the update data is the update data collected again after the collection configuration parameters are updated, and the collection operation of the update data is simple. It should be noted that after completing the collection of the update data, it is also necessary to perform a reporting trigger evaluation on the update data to determine whether the update data is reported to the target platform, and the reporting trigger evaluation of the update data is similar to the reporting trigger evaluation operation of the target data, which will not be repeated here.

[0146] In steps S801 to S804 shown in this embodiment, when it is necessary to collect other data of the target front-end web page, it is only necessary to update the data collection configuration parameters, making the collection operation of updating data easier and more convenient. There is no need to manually modify the code and then reload it, which saves manpower for data collection and improves the efficiency of updating data collection.

[0147] Please refer to Fig. 9 In the embodiment of the present application, the collection page information of the target front-end web page is set in the collection page configuration item on the data collection platform, and the collection mode information of the target front-end web page is set in the collection mode configuration item on the data collection platform, and then the data collection platform generates a point-burying data packet. The point-burying data packet is introduced and registered into the server where the target front-end web page is located, and the point-burying data packet is loaded through the server to generate three target data collection units. Among them, the three target data collection units are defined as "moudleLink", "modulePage" and "moduleSeesion" respectively. Then, the three target data collection units configure the collection mode and web page information, as well as the platform address for reporting data according to the configuration file in the point-burying data packet. At the same time, when it is determined that each original web page level of the target front-end web page has set up data to allow tracking, when the target front-end web page is clicked into by the user, the three target data units respectively collect the data generated by the corresponding web page level to obtain the target data. Finally, the target data is encrypted and automatically reported to the target platform in the priority order of moduleLink>modulePage>moduleSeesiond, realizing automatic data collection. The collection operation only needs to be configured in advance on the data collection platform, which not only saves manpower for data collection operations, but also improves the efficiency of data collection.

[0148] See also Fig.10 The present application also provides a data acquisition device that can implement the above data acquisition method. The device includes:

[0149] The parameter acquisition module 1001 is used to acquire the data collection configuration parameters of the target front-end web page; wherein the data collection configuration parameters include: collection web page information and collection mode information;

[0150] Configuration module 1002, used to configure and process the preset original acquisition module according to the acquired web page information and acquisition mode information to obtain a target acquisition module; wherein the target acquisition module includes at least one target data acquisition unit;

[0151] The information acquisition module 1003 is used to acquire the original web page hierarchy of the target front-end web page and the hierarchy setting information of the original web page hierarchy;

[0152] A level screening module 1004 is used to screen out a target web page level from the original web page level based on the level setting information;

[0153] The data collection module 1005 is used to collect data at the target web page level through each target data collection unit to obtain target data.

[0154] The specific implementation of the data acquisition device is basically the same as the specific implementation of the above-mentioned data acquisition method, and will not be repeated here.

[0155] The embodiment of the present application also provides a computer device, the computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the above data collection method when executing the computer program. The computer device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0156] See also Fig.11 , Fig.11 The hardware structure of a computer device according to another embodiment is shown, and the computer device includes:

[0157] The processor 1101 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0158] The memory 1102 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1102 can store an operating system and other applications. When the technical solution provided in the embodiments of this specification is implemented by software or firmware, the relevant program code is stored in the memory 1102, and the processor 1101 calls and executes the data acquisition method of the embodiments of this application;

[0159] Input / output interface 1103, used to implement information input and output;

[0160] The communication interface 1104 is used to realize the communication interaction between the device and other devices. The communication can be realized through a wired manner (such as USB, network cable, etc.) or a wireless manner (such as mobile network, WIFI, Bluetooth, etc.);

[0161] A bus 1105 that transmits information between various components of the device (e.g., the processor 1101, the memory 1102, the input / output interface 1103, and the communication interface 1104);

[0162] The processor 1101 , the memory 1102 , the input / output interface 1103 and the communication interface 1104 are connected to each other in communication within the device via the bus 1105 .

[0163] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program, and the computer program implements the above-mentioned data collection method when executed by a processor.

[0164] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0165] The data collection method and apparatus, computer equipment and storage medium provided in the embodiments of the present application configure the original collection module into a target collection module according to the collection web page information and collection mode information configured in the target front-end web page, select the target web page level that allows data collection from the original web page level of the target front-end web page, and finally complete the data collection of the target web page level through each target data collection unit in the target collection module. There is no need to configure the code for the components of the target front-end web page one by one in advance, which makes the data collection operation simpler, saves manpower, and improves the efficiency of data collection.

[0166] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0167] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0168] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0169] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0170] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0171] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0172] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0173] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0174] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0175] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0176] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A data collection method, characterized in that: The method comprises: Acquire data collection configuration parameters of the target front-end webpage; wherein the data collection configuration parameters include: collection webpage information and collection mode information; According to the collected web page information and the collected mode information, a preset original collection module is configured and processed to obtain a target collection module; wherein the target collection module includes at least one target data collection unit; Acquire the original webpage level of the target front-end webpage and level setting information of the original webpage level; Filtering a target web page hierarchy from the original web page hierarchy based on the hierarchy setting information; Each of the target data collection units collects data at the target web page level to obtain target data.

2. The method according to claim 1, characterized in that: The collected web page information includes: web page interface information, collection level information and collection attribute information, and the original collection module includes at least one original data collection unit; The configuring and processing the preset original acquisition module according to the acquired web page information and the acquisition mode information to obtain the target acquisition module includes: According to the webpage interface information and the collection level information, each of the raw data collection units is configured with a collection address to obtain a preliminary data collection unit; According to the collection attribute information, the preliminary data collection unit is configured with collection attributes to obtain a candidate data collection unit; Performing collection mode configuration on each of the candidate data collection units according to the collection mode information and the collection level information to obtain the target data collection unit; The target data acquisition units are spliced ​​to obtain the target acquisition module.

3. The method according to claim 2, characterized in that The step of configuring the collection mode of each of the candidate data collection units according to the collection mode information and the collection level information to obtain the target data collection unit includes: Acquire hierarchical function information of each level in the target front-end webpage according to the collected hierarchical information; Filtering target mode information from the acquisition mode information according to the hierarchical function information; The target data collection unit is obtained by configuring the collection mode of each candidate data collection unit according to the target mode information.

4. The method according to any one of claims 1 to 3, characterized in that: After each of the target data collection units collects data from the target web page level to obtain target data, the method further includes: Performing a reporting trigger evaluation on the target data according to a preset reporting trigger rule to obtain reporting trigger evaluation information; wherein the reporting trigger evaluation information indicates whether the target data is reported or not reported; If the reporting trigger evaluation information indicates that the target data is to be reported, the target data is reported to the target platform.

5. The method according to claim 4, characterized in that The reporting triggering rules include: triggering event information and triggering node information; The step of performing a reporting trigger evaluation on the target data according to the reporting trigger rule to obtain reporting trigger evaluation information includes: Analyze the target data to obtain data analysis information; Extracting target event information from the target data according to the data parsing information and the trigger node information; Compare the target event information with the trigger event information to obtain event comparison information; The target data is subjected to a reporting trigger evaluation according to the event comparison information to obtain the reporting trigger evaluation information.

6. The method according to claim 4, characterized in that If the reporting triggering evaluation information is characterized by reporting the target data, reporting the target data to the target platform includes: If the reporting trigger evaluation information indicates that the target data is to be reported, acquiring collection unit information of the target data; Encrypting the target data to obtain encrypted data; Prioritize the encrypted data according to the acquisition unit information to obtain data priority information; The encrypted data is reported to the target platform according to the data priority information.

7. The method according to any one of claims 1 to 3, characterized in that: After each of the target data collection units collects data from the target web page level to obtain target data, the method further includes: receiving configuration update requests; Update the data collection configuration parameters according to the configuration update request to obtain updated collection configuration parameters; According to the updated acquisition configuration parameters, the target data acquisition unit is configured and updated to obtain an updated data acquisition unit; The update data collection unit collects data from the target front-end webpage to obtain update data.

8. A data acquisition device, characterized in that: The device comprises: The parameter acquisition module is used to acquire the data acquisition configuration parameters of the target front-end webpage; wherein the data acquisition configuration parameters include: acquisition webpage information and acquisition mode information; A configuration module, configured to configure a preset original acquisition module according to the acquisition webpage information and the acquisition mode information to obtain a target acquisition module; wherein the target acquisition module includes at least one target data acquisition unit; An information acquisition module, used to acquire the original webpage level of the target front-end webpage and level setting information of the original webpage level; A level screening module, used for screening out a target web page level from the original web page level based on the level setting information; The data collection module is used to collect data from the target web page level through each of the target data collection units to obtain target data.

9. A computer device, characterized in that: The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the data collection method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the data collection method according to any one of claims 1 to 7 is implemented.

Images