Buried point data processing method, system and device for APP product and medium

Abstract

The invention discloses a buried point data processing method, system and device for an APP product and a medium, and the method specifically comprises the steps: intercepting all network request information of the APP product according to a man-in-the-middle agent tool, judging whether each piece of network request information is buried point request information or not, and if the network request information is the buried point request information, executing the step 1; if so, forwarding the burying point request information to a Django server; based on a preset data cleaning rule, performing structured processing on the burying point request information according to the Django server to obtain target burying point data; and forwarding the target burying point data to a front-end page based on a websocket communication technology, and displaying the target burying point data in real time on the front-end page. According to the method, full-process automatic acquisition, analysis, verification and display of the burying point data are realized, and the efficiency and accuracy of mobile App burying point testing are greatly improved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a method, system, device and medium for processing embedded data for APP products. Background Art

[0002] In the current mobile application development process, the collection and analysis of tracking data is an important means to evaluate product performance, user behavior and optimize product experience. However, there are some common problems with the tracking data reported by existing mobile apps, which not only affect the quality of the data, but also greatly increase the workload of testers.

[0003] Specifically, existing tracking data is usually reported in an unstructured data format, which contains a lot of redundant information and raw noise data. The existence of these redundant and noisy data not only increases the difficulty of data processing, but also may lead to inaccurate data analysis results. For testers, if they use traditional packet capture tools (such as Charles) to perform package testing, they need to spend a lot of time and energy to screen and verify valid tracking data. The test time cycle is long, the work is highly repetitive, and the test content is cumbersome.

[0004] In addition, since tracking codes are usually manually inserted by developers, it is easy to cause tracking errors due to code modifications or new features during product version iterations. Such errors not only affect the accuracy of the data, but may also mislead product decisions. Therefore, in each version iteration cycle, testers need to spend a certain amount of time to perform regression testing on tracking data to ensure that the tracking data of the new version meets the expected results. However, traditional regression testing methods are not only inefficient, but also prone to missing errors, and cannot meet the needs of fast-iteration product development. Summary of the invention

[0005] The purpose of the present invention is to provide a method, system, device and medium for processing tracking data of APP products, which realizes the full-process automated collection, analysis, verification and display of tracking data, greatly improves the efficiency and accuracy of mobile App tracking testing, and solves at least one of the above-mentioned prior art problems.

[0006] In a first aspect, the present invention provides a method for processing buried data for an APP product, the method specifically comprising:

[0007] Intercept all network request information of APP products through the middleman proxy tool, and determine whether each network request information is a buried request information. If it is a buried request information, forward the buried request information to the Django server;

[0008] Based on the preset data cleaning rules, the Django server performs structured processing on the tracking request information to obtain the target tracking data;

[0009] Based on the websocket communication technology, the target burying point data is forwarded to the front-end page, and the target burying point data is displayed in real time on the front-end page.

[0010] In a second aspect, the present invention provides a data processing system for APP products, the system specifically comprising:

[0011] The first processing module is used to intercept all network request information of the APP product according to the middleman proxy tool, and determine whether each network request information is a buried request information. If it is a buried request information, the buried request information is forwarded to the Django server;

[0012] The second processing module is used to perform structured processing on the tracking request information according to the Django server based on the preset data cleaning rules to obtain the target tracking data;

[0013] The third processing module is used to forward the target burying point data to the front-end page based on the websocket communication technology, and display the target burying point data in real time on the front-end page.

[0014] In a third aspect, the present invention provides a computer device, comprising: a memory and a processor and a computer program stored in the memory. When the computer program is executed on the processor, a method for processing buried data for an APP product as described in any one of the above methods is implemented.

[0015] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the method for processing buried point data for an APP product as described in any one of the above methods is implemented.

[0016] Compared with the prior art, the present invention has at least one of the following technical effects:

[0017] 1. The present invention realizes the full-process automated collection, analysis, verification and display of tracking data, greatly improving the efficiency and accuracy of mobile App tracking testing.

[0018] 2. The present invention intercepts and filters the tracking request information through the middleman proxy tool, effectively avoiding the interference of non-tracking data and improving the pertinence and efficiency of data processing.

[0019] 3. The present invention forwards the screened embedding request information to the Django server for further processing, thereby realizing centralized management and efficient use of data.

[0020] 4. The present invention forwards the processed target buried point data to the front-end page in real time based on websocket communication technology, so that relevant personnel can monitor data changes in real time and respond in time.

[0021] 5. The present invention accurately determines whether network request information is a point-of-sale request through preset point-of-sale request feature rules (including URL features, request frequency features and source IP features), and uses an SSL encrypted channel to transmit the point-of-sale request information, thereby improving the accuracy and reliability of the judgment, ensuring the security of data transmission, and preventing data leakage and tampering.

[0022] 6. The present invention adopts regular expression matching and removes redundant parameters, performs noise filtering on the first buried point data according to the Gaussian distribution noise data model, extracts key parameters through preset mapping rules and forms target buried point data, effectively reduces data redundancy, improves data availability and accuracy, and makes data more structured, easy to analyze and use.

[0023] 7. The present invention classifies the target buried data into business categories according to the naive Bayes classification model, adopts the K-Means clustering algorithm to perform cluster analysis on user attribute characteristics, and displays the buried data in real time on the front-end page through a visualization component, making the data more orderly and easy to understand, which helps to discover the characteristics and needs of user groups, provides strong support for product optimization, and enables relevant personnel to intuitively understand data changes and business trends.

[0024] 8. The present invention labels and classifies target buried point data, and dynamically adjusts the access scope based on the role-based access control model, thereby ensuring the security and compliance of the data and meeting the data needs of different personnel.

[0025] 9. The present invention optimizes regression test cases through K-means clustering algorithm, reduces redundancy and repeated testing, and improves test efficiency and quality.

[0026] 10. The present invention uses the Appium automated testing tool to perform regression testing, ensuring that the target tracking data of the new version meets the expected results and reducing the risk of product release.

[0027] 11. The present invention verifies the correctness of the decoding parameter values ​​based on the preset buried data verification rules, performs statistical analysis on the verification effect, and sets the verification failure rate threshold to trigger an abnormal alarm, which helps to discover and solve problems in a timely manner and ensure the stability and reliability of the data. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0029] Figure 1 It is a flowchart of a method for processing buried point data for an APP product provided by the first embodiment of the present invention;

[0030] Figure 2 It is a flow chart of a method for processing buried point data for an APP product provided by the second embodiment of the present invention;

[0031] Figure 3 It is a flowchart of a method for processing buried point data for an APP product provided by the third embodiment of the present invention;

[0032] Figure 4 It is a flowchart of a method for processing buried point data for an APP product provided by a fourth embodiment of the present invention;

[0033] Figure 5 It is a flowchart of a method for processing buried point data for an APP product provided by the fifth embodiment of the present invention;

[0034] Figure 6 It is a flowchart of a method for processing buried point data for an APP product provided by a sixth embodiment of the present invention;

[0035] Figure 7 It is a flowchart of a method for processing buried point data for an APP product provided by the seventh embodiment of the present invention;

[0036] Figure 8 It is a flowchart of a method for processing buried point data for an APP product provided by an eighth embodiment of the present invention;

[0037] Fig. 9 It is a structural diagram of a tracking point data processing system for APP products provided by an embodiment of the present invention;

[0038] Fig.10 It is a structural schematic diagram of a computer device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0039] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0040] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0041] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0042] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.

[0043] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0044] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0045] In the embodiment of the present application, the execution subject of the process includes a terminal device, which includes but is not limited to: a server, a computer, a smart phone, a tablet computer, and other devices capable of executing the method disclosed in the present application. Figure 1 A flow chart of a method for processing buried data for an APP product disclosed in the first embodiment of the present invention is shown, and the details are as follows:

[0046] S101, intercept all network request information of APP products according to the middleman proxy tool, and determine whether each network request information is a buried request information. If it is a buried request information, forward the buried request information to the Django server.

[0047] In this embodiment, a middleman proxy tool such as Charles or mitmproxy is selected to intercept all network request information of the APP product, set up a Django server to receive and process the buried request information, and select the APP product to be tested. It is necessary to ensure that its network request can be intercepted by the middleman proxy tool. When the user operates the APP, the APP will send a network request to the server. The middleman proxy tool will intercept these network requests and display them in the tool interface. According to the preset buried request feature rules, it is determined whether each network request information is a buried request information. If a certain network request information is determined to be a buried request information, it is transmitted to the Django server through a secure channel. After receiving the buried request information, the Django server performs further processing and analysis.

[0048] In this embodiment, the middleman proxy tool can efficiently intercept all network request information of the APP product, and quickly determine which requests are buried requests according to preset rules, avoiding interference from non-buried data and improving the pertinence and efficiency of data processing. The buried request information is transmitted through the SSL encrypted channel to ensure the security of data transmission, prevent data from being leaked or tampered with during transmission, and ensure the integrity and credibility of the data.

[0049] S102, based on the preset data cleaning rules, the Django server performs structured processing on the tracking request information to obtain the target tracking data.

[0050] In this embodiment, the data cleaning rules may include: removing invalid data, deleting the tracking request information with incorrect format, incomplete content or unparseable; outlier processing, for numerical data, setting a reasonable threshold, replacing or deleting the outliers that exceed the threshold; missing value filling, for missing key fields, filling according to the context or preset rules, such as using default values, mean or median, etc.; data format unification, converting data from different sources and different formats into a unified format for subsequent processing and analysis. The Django server receives the tracking request information from the middleman proxy tool through a specific interface, and performs preliminary analysis and verification on the received tracking request information to ensure the integrity and accuracy of the data. According to the preset data cleaning rules, the data is cleaned and preprocessed. The cleaned data is structured and converted into a format that is easy to analyze and store. The data conversion operations involved include data type conversion, data splitting, data merging, etc. The target tracking data after structured processing is stored in a database, such as MySQL, PostgreSQL, etc. At the same time, a data query interface is provided to allow other systems or modules to query and analyze the target tracking data as needed.

[0051] In this embodiment, the preset data cleaning rules are used to effectively remove invalid data, process abnormal values ​​and missing values, improve the accuracy and completeness of the data, and structured processing makes the data more standardized, easy to understand and analyze. The Django server can quickly receive, clean and structure the buried request information through automated processing procedures, which reduces the possibility of manual intervention and errors and improves processing efficiency.

[0052] S103, based on the websocket communication technology, forward the target burying point data to the front-end page, and display the target burying point data in real time on the front-end page.

[0053] In this embodiment, the Django server listens to the WebSocket connection request sent by the front-end page. The Django server performs a legitimacy check on the connection request, for example, verifying the source IP, Token, etc. After the check is passed, a WebSocket persistent connection is established and the connection status is recorded. The Django server pushes the structured buried data to the front-end page through the established WebSocket connection. The front-end page receives the data pushed by the back-end and returns the receiving status. If the receiving status is normal, the built-in JSON in JavaScript is used. The parse method parses the data in JSON format. After successful parsing, the pre-defined HTML template (including table structure and CSS style) is used to render the parsed buried data into an HTML table, and the rendered buried data is displayed in real time on the front-end page, the page content is updated, and real-time visualization of the buried data is achieved.

[0054] In this embodiment, the real-time forwarding and display of target buried data is realized through WebSocket communication technology, and users can instantly see the latest data changes on the front-end page. Real-time display of target buried data improves user experience and interactivity, and users can intuitively understand data changes and business trends through the front-end page.

[0055] As an example of application of the above steps S101 to S103, refer to Figure 2 , use the mitmproxy middleman interception framework to intercept the request of the test phone, obtain the tracking request information of the App product and then forward it to the Django server. When intercepting the request, it only obtains the request information, does not interrupt the normal request, and does not affect the normal request process; after the mitmproxy middleman interception framework confirms that a certain request information is a tracking request, it forwards the tracking request to the Django server. After the Django server receives the original tracking data, it structures the data, maps the original parameter values ​​in the data, decodes each transcoded parameter value, and other operations to parse the original data into the most intuitive tracking data; the parsed data is sent to the WebSocket channel of the tracking. Since there are many tracking requests, forwarding data through the WebSocket connection can save service resources and the front end can receive the tracking data more in real time. The front end receives the data and renders it through a beautiful table.

[0056] In some embodiments, Figure 3 As shown, the process of determining whether each network request information is a buried request information, and if it is a buried request information, forwarding the buried request information to the Django server specifically includes:

[0057] S1011, judging whether each network request information is a tracking request information according to a preset tracking request feature rule, wherein the tracking request feature rule includes a URL feature, a request frequency feature, and a source IP feature;

[0058] S1012: If the network request information is a embedding request information, the embedding request information is transmitted to the Django server through an SSL encrypted channel.

[0059] In this embodiment, according to the preset tracking request feature rules, the URL feature, request frequency feature and source IP feature information of each network request are obtained. The obtained network request feature information is analyzed by a machine learning algorithm, such as a decision tree, a support vector machine or a neural network, to determine whether the network request meets the tracking request feature. If the judgment result is that the network request meets the tracking request feature, the network request is marked as a tracking request, and the key information of the request, such as request parameters, request headers, etc., is extracted. The extracted tracking request key information is encrypted, and an SSL encryption channel is used to ensure the security of the data transmission process. The encrypted tracking request information is transmitted to the Django server via the network. After receiving the encrypted tracking request information, the Django server performs decryption processing to obtain the original tracking request data. According to the characteristics of the tracking request data and business needs, corresponding data analysis and processing are performed, such as user behavior analysis, abnormal request detection, etc., to provide data support for subsequent business decisions.

[0060] Exemplarily, first, the identification of tracking requests is based on preset feature rules, which usually include URL features, request frequency features, and source IP features. For example, a typical tracking request may have a URL with a specific pattern, such as a path containing " / track_event". In addition, the frequency of tracking requests is usually high because multiple operations of each user may trigger tracking. The source IP feature helps to identify whether the request comes from a legitimate user or a centralized server. Through the combination of these features, the system can effectively identify which requests are tracking requests. Once the tracking request is identified, the next key step is to ensure the secure transmission of this data. Here, the SSL encrypted channel plays an important role. SSL (Secure Sockets Layer) is a security protocol used to establish encrypted links on the Internet to ensure the security and integrity of data during transmission. For example, when tracking data is transmitted from the client to the server, SSL encryption can prevent man-in-the-middle attacks and ensure that sensitive information such as user behavior data is not leaked. After the data is transmitted to the Django server, the server will process the data and perform subsequent operations such as storage and analysis. Django is an advanced Python web framework that supports rapid development and clean, practical design. With Django, developers can easily build database-driven websites, which is especially important for processing large amounts of buried data. The server cleans, aggregates, and analyzes the data to provide in-depth insights into user behavior and support business decisions. This process not only allows efficient processing and analysis of user data, but also ensures the security and reliability of the data processing process. This is crucial for modern network services that rely on user behavior analysis.

[0061] In some embodiments, Figure 4 As shown, based on the preset data cleaning rules, the Django server performs structured processing on the tracking request information to obtain the target tracking data, which specifically includes:

[0062] S1021, using a regular expression to match redundant parameters in the tracking request information, removing the redundant parameters, and obtaining first tracking data;

[0063] S1022, performing noise filtering on the first buried point data according to a preset Gaussian distribution noise data model to obtain second buried point data;

[0064] S1023, according to the preset mapping rules, key parameters are extracted from the second buried point data to form target buried point data, and the mapping rules include the correspondence between parameter names and key names, and the type and format requirements of parameter values.

[0065] In this embodiment, according to the characteristics of the buried point request information, a regular expression matching rule is preset to identify the redundant parameters therein; the buried point request information is matched using the regular expression matching rule, and the matched redundant parameters are removed to obtain the first buried point data; according to the statistical characteristics of the buried point request information and the Gaussian distribution noise data model, the algorithm parameters for noise filtering the first buried point data are determined; the first buried point data is subjected to noise filtering processing using the algorithm parameters to obtain the buried point data after preliminary filtering; it is determined whether the noise level of the buried point data after preliminary filtering is lower than a preset threshold value, and if so, the buried point data after preliminary filtering is determined as the second buried point data; if the noise level of the buried point data after preliminary filtering is higher than or equal to the preset threshold value, the algorithm parameters are adjusted, and the noise filtering processing is returned to be executed until the noise level is lower than the preset threshold value.

[0066] According to the business attributes of the tracking request information, a data cleaning rule library is established to clean the second tracking data. According to the preset mapping rules, the second tracking data summary is obtained, and the parameter name and parameter value are extracted for each tracking data in the summary. It is determined whether the extracted parameter name has a corresponding key name in the mapping rule. If so, the parameter type and format requirements corresponding to the key name are obtained. According to the parameter type and format requirements, the extracted parameter value is type converted and formatted to obtain a standardized parameter value. The parameter name is used as the key name and the standardized parameter value is used as the key value to construct a key-value pair to form a key parameter. The key parameters are arranged in the order in the mapping rules to generate a parameter sequence that conforms to the target tracking data format. The parameter sequence is encoded to obtain the compressed target tracking data. The target tracking data is stored in the preset database and an index is established for subsequent quick query and analysis.

[0067] Exemplarily, in network data analysis, it is first necessary to clean up the redundant parameters in the buried point request information. For example, the user behavior buried point information of an e-commerce website may contain information such as the user's click time, click location, product ID, etc., but it may also contain some unnecessary parameters such as page background color. By using regular expressions, these irrelevant parameters can be effectively identified and removed, thereby obtaining more concise and useful first buried point data. The implementation of this step helps to improve the efficiency and accuracy of data processing. Next, it is necessary to filter the first buried point data for noise. In practical applications, data is often affected by various noises, such as system errors, environmental interference, etc. The use of Gaussian distribution noise data model can effectively simulate these noises and remove or reduce the impact of noise through corresponding mathematical methods. For example, if a buried point data shows that a user frequently clicks on the same product in a short period of time, this may be caused by system errors or user misoperation. By analyzing the distribution of these data and comparing them with the Gaussian model, it is possible to identify which data is abnormal, thereby obtaining more accurate second buried point data. Finally, extracting key parameters according to preset mapping rules is a key step in forming target buried point data. The mapping rules include the correspondence between parameter names and key names, as well as the type and format requirements of parameter values, which ensures the standardization and consistency of data. For example, "product ID" is mapped to "product_id", and its value type is specified as an integer. This not only helps with subsequent data analysis and processing, but also facilitates data storage and transmission. Through this series of processing, it can ensure that valuable data is extracted from the original network request information to provide support for subsequent data analysis and business decisions. The implementation of these steps not only improves the efficiency of data processing, but also ensures data quality, providing a solid foundation for subsequent data analysis and application. In this way, the development and optimization of the business can be effectively supported, bringing obvious technical effects and commercial value.

[0068] In some embodiments, Figure 5 As shown, the target embedding data is displayed in real time on the front-end page, specifically including:

[0069] S1031, classifying the target buried point data according to the business category according to a preset naive Bayes classification model to obtain target buried point data of different business categories;

[0070] S1032, performing cluster analysis on target embedding data of different business categories using a K-Means clustering algorithm according to the attribute characteristics of the user, and obtaining display embedding data including user grouping attributes and business classification attributes;

[0071] S1033, display the display embedding data in real time on the front-end page through a visualization component.

[0072] In this embodiment, according to a preset naive Bayes classification model, target buried point data are obtained, and the classification model is used to perform classification processing on the business category characteristics of the target buried point data to obtain target buried point data subsets of different business categories; for each target buried point data subset obtained by the classification process, its data characteristics are extracted, and based on the data characteristics and preset business rules, the specific business category to which the subset belongs is determined, and if the rules of a certain business category are met, the subset is marked as the corresponding business category; for the target buried point data subset whose specific business category cannot be determined, it is sent to the manual review module, and the business category is determined manually, and the manual determination result is used as the business category label of the subset; according to the business category label of each target buried point data subset, the subsets of different business categories are summarized to obtain a set of target buried point data divided by business category.

[0073] The K-Means clustering algorithm is used to perform cluster analysis on the target embedding data. By calculating the similarity between data points, similar data points are divided into the same cluster; the user grouping attributes are obtained based on the clustering results, and users with similar attribute characteristics are divided into corresponding groups; combined with the business classification attributes, user groups of different business categories are associated to obtain business-related user group divisions; for different user groups, corresponding display embedding data are generated to record the user's behavior trajectory in different business scenarios; by analyzing the display embedding data, the behavior patterns and preference characteristics of different user groups in business scenarios are explored; based on user grouping attributes and business classification attributes, a user portrait model is constructed to provide data support for personalized recommendations and precision marketing.

[0074] By embedding data collection code in the front-end page, the user's interactive events and behavior data, such as clicks, scrolling, page visits, etc., are captured and sent to the server for storage and processing. Data preprocessing: Clean, filter and convert the collected data, remove invalid or abnormal data, and ensure the accuracy and completeness of the data. According to business needs, classify, aggregate and calculate the data to generate a data format that meets the requirements of visual display.

[0075] Build a visualization component library: Based on front-end frameworks (such as Vue, React, etc.) and visualization libraries (such as ECharts, D3.js, etc.), develop a flexible and reusable visualization component library, including various chart types (such as line charts, bar charts, pie charts, etc.) and interactive functions (such as zooming, filtering, hovering, etc.).

[0076] Design visualization layout: Design a reasonable visualization layout and interface based on the characteristics of the embedded data and business needs, arrange the position, size and style of the chart reasonably, provide clear and intuitive data display and interaction methods, and facilitate users to quickly understand and analyze data.

[0077] Real-time data update: Establish a real-time communication mechanism (such as WebSocket) between the front-end and the server. When the server receives new tracking data, it pushes the data to the front-end in a timely manner, triggering the data update and re-rendering of the visualization component to achieve real-time display of data.

[0078] Performance optimization: Use data lazy loading, component lazy loading and other technologies to optimize the loading speed and rendering performance of visualization components. For large-scale buried data, you can use fragmented loading, data compression and other methods to reduce the volume and time of data transmission.

[0079] Interaction and feedback: Add interactive functions to visualization components, such as data filtering, time range selection, data drilling, etc., to facilitate users to explore and analyze data. At the same time, provide user feedback and exception reporting mechanisms to promptly discover and solve problems in the visualization process and continuously optimize the user experience.

[0080] Exemplarily, first, the target buried point data is classified and processed by the naive Bayes classification model, and the data can be effectively divided by business category. For example, assuming that an e-commerce platform has three main business categories of clothing, electronic products and home furnishings, the user behavior data can be classified into these three categories by analyzing the user's purchasing behavior and browsing habits through the naive Bayes model. This classification process is based on a probability model and takes into account the independence between each feature, so that the processing process is simple and efficient. Next, the K-Means clustering algorithm is used to perform further cluster analysis on the target buried point data of each business category. Taking the clothing business as an example, users can be grouped according to attribute characteristics such as user age, purchase frequency, price preference, etc., so as to understand the specific needs of different user groups more finely. The K-Means algorithm calculates the distance between the data point and each cluster center and iteratively updates the cluster center until the optimal clustering result appears. In this way, it can not only reveal the hidden needs of users, but also help merchants formulate more targeted marketing strategies. Finally, the obtained display buried point data is displayed in real time through the visualization component of the front-end page. For example, a dynamic dashboard can be designed to display the purchasing power, activity, and preference changes of different user groups in real time. This visualization not only makes the data results intuitive and easy to understand, but also helps business personnel quickly grasp market dynamics and user trends, so as to make more effective business decisions. Through this series of data processing and analysis, e-commerce platforms can better understand and serve their users, while also improving their market competitiveness.

[0081] In some embodiments, Figure 6 As shown, the method also includes:

[0082] S201, labeling and classifying the target buried point data to obtain target buried point data at different granularities;

[0083] S202, a role-based access control model dynamically adjusts the access scope of each access personnel to target tracking data at different granularities according to the roles and permissions of different access personnel.

[0084] In this embodiment, the original target buried point data is obtained, and the data is hierarchically annotated according to the preset hierarchical rules and annotation system to obtain data subsets with different granularity characteristics. For the annotated target buried point data, a clustering algorithm is used to classify the data, and the data with similar granularity is divided into the same data set to form a data set with obvious granularity differences. A role-based access control model is constructed, and the permission levels corresponding to different roles are pre-defined, and the permissions are mapped and associated with the data granularity. When an access request arrives, the role information of the visitor is obtained, and the permission level it possesses is determined by querying the RBAC model. According to the permission level of the visitor, the data set of the corresponding granularity is matched to generate the data range that the role can access. The visitor's accessible data range is compared with the data requested by the visitor, the visibility of the data is dynamically adjusted, and the data that is not allowed to be accessed is filtered out. Finally, the target buried point data that meets the visitor's role and permission is returned to realize dynamic access control based on the RBAC model.

[0085] For example, first of all, labeling these data means labeling the data with specific labels, such as "click the purchase button", "browse the product details page", etc., which is helpful for subsequent data analysis and understanding of user behavior. Next, classification processing is to divide the labeled data into different categories according to business needs. For example, the data can be divided into three major categories: "transaction behavior", "browsing behavior" and "search behavior". Such classification not only helps to analyze user behavior more systematically, but also to formulate more targeted marketing strategies according to different categories. In terms of data access management, the role-based access control model (RBAC) is an effective data security management strategy. The model dynamically adjusts the scope of access to data according to the user's role and authority. For example, for an e-commerce platform, the following roles can be set: ordinary employees, data analysts and senior managers. Ordinary employees may only have access to basic user behavior data, data analysts can access all classified point-of-sale data, and senior managers have the right to access all data including sensitive data. Through this role allocation and authority management, not only can the security of data be protected, but also the efficiency of data use can be ensured. For example, when data analysts conduct market trend analysis, they can access all the data they need without having to apply for permissions every time, which greatly improves work efficiency and the flexibility of data use. In addition, this role-based data access control also helps maintain data integrity and consistency. Because the access of each role is strictly controlled and managed, the risk of unauthorized access or erroneous operation of data can be effectively avoided. Through such a data management strategy, e-commerce platforms can not only protect user data security more effectively, but also provide customized data access services according to the business needs of different roles, thereby supporting more refined business operations and decision-making.

[0086] In some embodiments, Figure 7 As shown, the method also includes:

[0087] S301, obtaining target tracking data of different versions of the APP product, performing comparative analysis on the target tracking data of different versions, and extracting difference contents;

[0088] S302, generating regression test cases according to the difference content, classifying similar regression test cases using a K-means clustering algorithm, removing redundant regression test cases, and obtaining optimized regression test cases;

[0089] S303, according to the optimized regression test case, use the Appium automated testing tool to perform regression testing to determine whether the target tracking data of the new version meets the expected effect. If it does not meet the expected effect, generate a defect report by parsing the test log and error information.

[0090] In this embodiment, different versions of the target APP are obtained, and the buried data is collected for each version, and the buried data is stored by version. The buried data of different versions are compared, and the difference data between the versions are found through the data analysis algorithm, and the difference content set is extracted. According to each difference item in the difference content set, the corresponding regression test case is generated by using the automated test case generation technology. The generated regression test case is feature extracted to obtain the use case feature vector, and the similarity between the use cases is determined by calculating the similarity between the feature vectors. The K-means clustering algorithm is used to automatically classify similar regression test cases with the use case similarity as the distance metric, and similar use cases are divided into the same cluster. In each cluster, redundant regression test cases are removed through the use case screening strategy to obtain an optimized use case set, forming a regression test set after cluster optimization. Regression testing is performed through the Appium automated testing tool, and the obtained target buried data is compared with the expected effect to determine whether the expected effect is met. If the target buried data does not meet the expected effect, the test log and error information generated during the test are parsed. Based on the parsed test logs and error information, natural language processing technology is used to extract keywords and key phrases to determine the defect type and defect description. Based on the determined defect type and defect description, combined with the preset defect report template, a defect report that conforms to the standard format is automatically generated. The automatically generated defect report is saved to the specified defect management system, and the defect report is assigned to the relevant responsible person for subsequent processing through the system interface. Based on the processing results of the defect report, a machine learning algorithm is used to further optimize the regression test cases to improve the efficiency and accuracy of regression testing.

[0091] For example, first, by collecting target tracking data of different versions of APP products, users' specific operations in the APP, such as clicks, slides, etc., can be effectively tracked and recorded. For example, an e-commerce APP may adjust the layout of the product details page in different versions. By comparing the tracking data of "clicking the purchase button" in the new and old versions, changes in user behavior can be observed, thereby analyzing the impact of the new layout on user operations. Next, based on the difference in the content of these tracking data, targeted regression test cases can be generated. For example, if the click rate of the "add to cart" button in the new version is found to be abnormally reduced, a test case may need to be generated to verify whether there is a problem with this function. In order to improve the test efficiency, it is an efficient method to classify similar regression test cases using the K-means clustering algorithm. By calculating the feature distance between test cases and clustering test cases with similar test targets together, redundant test cases with high duplication or similarity can be effectively removed, thereby optimizing the test process. The optimized regression test cases will be used in the Appium automated testing tool to perform regression testing. Appium is an automation framework widely used in mobile application testing. It supports two major platforms, iOS and Android, and can simulate users' actual operations in the APP, such as clicks, slides, etc. Through automated testing, it is possible to quickly verify whether the target tracking data of the new version meets the expected effect. For example, if the test results show that the "payment" function in the new version cannot be completed normally, this will directly trigger the generation of a defect report. Finally, by parsing the test logs and error information, a detailed defect report can be generated. These reports not only record the specific circumstances of the problem in detail, but may also include environmental information and possible cause analysis of the error, providing the development team with an important basis for fixing the error. For example, if the log shows that the "payment failure" error frequently occurs in the case of poor network connection, the development team may need to optimize the processing logic of network requests to improve the performance of the APP in an unstable network environment. Through this series of refined data analysis and automated testing processes, not only can the quality of APP products be ensured, but also the efficiency of development and testing can be greatly improved, the time to market can be shortened, and ultimately user satisfaction and market competitiveness can be improved.

[0092] In some embodiments, Figure 8 As shown, the method also includes:

[0093] S401, obtaining an encrypted parameter value in the target buried point data, and decoding the encrypted parameter value using a Base64 decoding algorithm to obtain a decoded parameter value;

[0094] S402, based on a preset embedded data verification rule, the decoding parameter value is verified for correctness to obtain a verification result, wherein the embedded data verification rule includes a data type verification, a value range verification, and a parameter combination logic verification;

[0095] S403, performing statistical analysis on the verification results to obtain a verification pass rate and a verification failure rate;

[0096] S404: Determine whether to trigger an abnormal alarm by comparing the verification failure rate with a preset failure rate threshold.

[0097] In this embodiment, the encrypted parameter value in the target buried data is obtained, and the encrypted parameter value is decoded using the Base64 decoding algorithm to obtain the decoded parameter value. According to the preset buried data verification rules, the data type verification is performed on the decoded parameter value to determine whether the data type of the decoded parameter value meets the expected type. If not, the parameter value is marked as failed to verify. For the decoded parameter value that passes the data type verification, the value range verification is further performed to determine whether the parameter value is within the preset value range. If it exceeds the value range, the parameter value is marked as failed to verify. For the decoded parameter value that passes the data type verification and value range verification, according to the preset parameter combination logic rules, it is determined whether the combination relationship between different parameters conforms to the business logic. If not, the relevant parameters are marked as failed to verify. The results of the data type verification, value range verification and parameter combination logic verification are comprehensively considered to determine the final verification result of each decoded parameter value, and the parameter value that passes the verification is subjected to subsequent business processing. For the parameter value that fails the verification, it is decided whether to perform exception processing or ignore the parameter according to the business importance and abnormal situation of the parameter to ensure the normal progress of the main business process.

[0098] Obtain verification result data, perform statistics on the verification results, calculate the number of verification passes and the number of verification failures; calculate the verification pass rate based on the number of verification passes and the total number of verifications; calculate the verification failure rate based on the number of verification failures and the total number of verifications; obtain the preset verification failure rate threshold, and compare the calculated verification failure rate with the preset threshold; if the verification failure rate is greater than the preset threshold, it is judged as a verification anomaly and an anomaly alarm is triggered; use data visualization technology to generate a verification result statistical analysis report to intuitively display the verification pass rate and failure rate; establish a verification anomaly prediction model through machine learning algorithms such as decision trees or support vector machines, and train it based on historical verification data; input the current verification result data into the prediction model to determine whether verification anomalies are likely to occur, issue early warnings, and improve verification efficiency and accuracy.

[0099] For example, when performing data analysis on APP products, it is first necessary to process the encrypted parameter values ​​in the target buried data. For example, the shopping cart function of an e-commerce APP may encrypt the user's product ID and price information to protect the user's data security. These encrypted parameter values ​​are usually encoded using Base64, so they need to be decoded using the Base64 decoding algorithm to obtain the original product ID and price information. After decoding, the next step is to verify the correctness of these parameter values. The verification rules can include multiple aspects, such as data type verification to ensure that the product ID is an integer and the price is a floating point number; value range verification to ensure that the product ID is in the valid product inventory list, the price is not less than zero and does not exceed the highest selling price of the product; parameter combination logic verification ensures that the combination of product ID and price is reasonable in business logic, for example, a product with a specific ID cannot be lower than a certain price bottom line. Through these verifications, the verification pass rate and verification failure rate can be counted. For example, if there are 1,000 parameter values ​​in a test and 950 of them pass the verification, then the verification pass rate is 95% and the verification failure rate is 5%. These statistical results are crucial for evaluating the quality of data processing. Finally, by comparing the verification failure rate with the preset failure rate threshold, it can be determined whether an abnormal alarm needs to be triggered. Assuming that the preset failure rate threshold is 3%, if the actual verification failure rate is 5%, it exceeds the threshold, and the system should trigger an abnormal alarm to prompt relevant personnel to conduct further inspection and processing. This mechanism helps to discover problems in data processing in a timely manner and ensure the data quality and user experience of the APP.

[0100] Reference Fig. 9 An embodiment of the present invention provides a data processing system 9 for APP products, and the system 9 specifically includes:

[0101] The first processing module 901 is used to intercept all network request information of the APP product according to the middleman proxy tool, and determine whether each network request information is a buried request information. If it is a buried request information, forward the buried request information to the Django server;

[0102] The second processing module 902 is used to perform structured processing on the tracking request information according to the Django server based on the preset data cleaning rules to obtain the target tracking data;

[0103] The third processing module 903 is used to forward the target burying point data to the front-end page based on the websocket communication technology, and display the target burying point data in real time on the front-end page.

[0104] It is understandable that if Figure 1The contents of the embodiment of the method for processing the buried data of the APP product shown in the figure are applicable to the embodiment of the buried data processing system for the APP product. The functions specifically implemented by the embodiment of the buried data processing system for the APP product are the same as those in the embodiment of the method for processing the buried data of the APP product shown in the figure. Figure 1 The method for processing data of embedded points for APP products shown in the embodiment is the same as that of the embodiment of the invention, and the beneficial effects achieved are the same as those of the embodiment of the invention. Figure 1 The beneficial effects achieved by the embodiment of the buried data processing method for APP products shown are also the same.

[0105] It should be noted that the information interaction, execution process and other contents between the above-mentioned systems are based on the same concept as the embodiment of the method of the present invention. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0106] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the system can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0107] Reference Fig.10 The embodiment of the present invention further provides a computer device 10, comprising: a memory 1002 and a processor 1001 and a computer program 1003 stored in the memory 1002. When the computer program 1003 is executed on the processor 1001, a method for processing buried point data for an APP product as described in any one of the above methods is implemented.

[0108] The computer device 10 may be a computing device such as a desktop computer, a notebook, a PDA, or a cloud server. The computer device 10 may include, but is not limited to, a processor 1001 and a memory 1002. Those skilled in the art will appreciate that Fig.10 This is only an example of the computer device 10 and does not constitute a limitation on the computer device 10 . The computer device 10 may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, it may also include input and output devices, network access devices, etc.

[0109] The processor 1001 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0110] In some embodiments, the memory 1002 may be an internal storage unit of the computer device 10, such as a hard disk or memory of the computer device 10. In other embodiments, the memory 1002 may also be an external storage device of the computer device 10, such as a plug-in hard disk, a smart memory card (SmartMedia Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the computer device 10. Further, the memory 1002 may also include both an internal storage unit of the computer device 10 and an external storage device. The memory 1002 is used to store an operating system, an application program, a boot loader (Boot Loader), data, and other programs, such as the program code of the computer program, etc. The memory 1002 may also be used to temporarily store data that has been output or is to be output.

[0111] An embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the method for processing buried point data for an APP product as described in any one of the above methods is implemented.

[0112] In this embodiment, if the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, the steps of the above-mentioned various method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium can at least include: any entity or device that can carry the computer program code to the camera / terminal device, recording medium, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium. For example, a USB flash drive, a mobile hard disk, a disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0113] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0114] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0115] In the embodiments disclosed in the present application, it should be understood that the disclosed devices / terminal equipment and methods can be implemented in other ways. For example, the device / terminal equipment embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0116] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

Claims

1. A method for processing buried data for APP products, characterized in that: The method specifically comprises: Intercept all network request information of APP products through the middleman proxy tool, and determine whether each network request information is a buried request information. If it is a buried request information, forward the buried request information to the Django server; Based on the preset data cleaning rules, the Django server performs structured processing on the tracking request information to obtain the target tracking data; Based on the websocket communication technology, the target burying point data is forwarded to the front-end page, and the target burying point data is displayed in real time on the front-end page.

2. The method according to claim 1, characterized in that The step of determining whether each network request information is a buried request information, and if so, forwarding the buried request information to the Django server, specifically includes: According to the preset tracking request feature rules, determine whether each network request information is tracking request information, wherein the tracking request feature rules include URL features, request frequency features, and source IP features; If the network request information is a buried request information, the buried request information is transmitted to the Django server through an SSL encrypted channel.

3. The method according to claim 1, characterized in that The method of performing structured processing on the tracking request information based on the preset data cleaning rules according to the Django server to obtain the target tracking data specifically includes: Use regular expressions to match redundant parameters in the tracking request information, remove the redundant parameters, and obtain first tracking data; Perform noise filtering on the first buried point data according to a preset Gaussian distribution noise data model to obtain second buried point data; According to preset mapping rules, key parameters are extracted from the second buried point data to form target buried point data. The mapping rules include the correspondence between parameter names and key names, as well as the type and format requirements of parameter values.

4. The method according to claim 1, characterized in that: The real-time display of the target embedding data on the front-end page specifically includes: Classify the target buried point data according to the business category according to the preset naive Bayes classification model to obtain target buried point data of different business categories; According to the user's attribute characteristics, the K-Means clustering algorithm is used to perform cluster analysis on the target tracking data of different business categories to obtain display tracking data containing user group attributes and business classification attributes; The display embedding data is displayed in real time on the front-end page through a visualization component.

5. The method according to any one of claims 1 to 4, characterized in that The method further comprises: Labeling and classifying the target buried point data to obtain target buried point data at different granularities; The role-based access control model dynamically adjusts the access scope of each access person to target tracking data at different granularities according to the roles and permissions of different access persons.

6. The method according to any one of claims 1 to 4, characterized in that The method further comprises: Obtain target tracking data of different versions of the APP product, perform comparative analysis on the target tracking data of different versions, and extract the difference content; Generate regression test cases according to the difference content, classify similar regression test cases using K-means clustering algorithm, remove redundant regression test cases, and obtain optimized regression test cases; According to the optimized regression test cases, regression testing is performed using the Appium automated testing tool to determine whether the target tracking data of the new version meets the expected results. If not, a defect report is generated by parsing the test logs and error information.

7. The method according to any one of claims 1 to 4, characterized in that The method further comprises: Obtain the encrypted parameter value in the target buried point data, and decode the encrypted parameter value using a Base64 decoding algorithm to obtain a decoded parameter value; Based on the preset embedded data verification rules, the correctness of the decoding parameter value is verified to obtain a verification result, and the embedded data verification rules include data type verification, value range verification, and parameter combination logic verification; Performing statistical analysis on the verification results to obtain a verification pass rate and a verification failure rate; By comparing the verification failure rate with a preset failure rate threshold, it is determined whether to trigger an abnormal alarm.

8. A data processing system for APP products, characterized in that: The system specifically comprises: The first processing module is used to intercept all network request information of the APP product according to the middleman proxy tool, and determine whether each network request information is a buried request information. If it is a buried request information, the buried request information is forwarded to the Django server; The second processing module is used to perform structured processing on the tracking request information according to the Django server based on the preset data cleaning rules to obtain the target tracking data; The third processing module is used to forward the target burying point data to the front-end page based on the websocket communication technology, and display the target burying point data in real time on the front-end page.

9. A computer device, characterized in that: include: A memory and a processor and a computer program stored in the memory, when the computer program is executed on the processor, implements the method for processing buried point data for an APP product as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a processor, the method for processing buried point data for an APP product as described in any one of claims 1 to 7 is implemented.

Images