C-end user data intelligent management system based on large model

Abstract

The invention relates to a large model-based C-end user data intelligent management system, which relates to the technical field of big data, and comprises a data transmission module for partitioning C-end user data so as to calculate dynamic parameters of each data block; determining the emergency degree according to the dynamic parameter of each data block, and sorting and transmitting the data blocks according to the emergency degree; the generation module is used for receiving the data blocks, and generating thermodynamic data and mining historical behavior data by tracking a user browsing path, analyzing a jump relation and identifying lost nodes so as to record operation behavior data of the user at the client; and the data analysis module identifies and corrects errors in the user data and the operation behavior data, performs feature selection and data reduction, and analyzes the reduced data to obtain analyzed data. According to the invention, the user data value can be efficiently mined, and automatic and intelligent management of the data is realized.

Description

Technical Field

[0001] The present invention relates to the field of big data technology, and in particular to a C-end user data intelligent management system based on a big model. Background Art

[0002] With the rapid development and popularization of the Internet, C-end (consumer-end) user data has shown an explosive growth trend. These data contain valuable information such as users' various behavior trajectories, preferences, and needs on the Internet, which is of great significance for enterprises to understand market dynamics, optimize product services, and formulate marketing strategies.

[0003] Some traditional data processing methods can only process structured data, but have limited processing capabilities for unstructured data (such as text, images, videos, etc.), which leads to a large amount of valuable information being ignored or wasted. In addition, some traditional user behavior analysis is based on simple statistical methods and rules, which makes it difficult to deeply explore the potential patterns and relationships of user behavior. Summary of the invention

[0004] The technical problem to be solved by the present invention is to overcome the deficiencies of the prior art and provide a C-end user data intelligent management system based on a large model, which can efficiently mine the value of user data and realize automated and intelligent management of data.

[0005] In order to solve the above technical problems, the basic concept of the technical solution adopted by the present invention is: A C-end user data intelligent management system based on a large model, comprising: The data acquisition module starts from the initial URL, requests and parses the target site data, obtains the new URL after deduplication and filtering, and puts it into the queue for cyclic crawling to obtain C-end user data; The data transmission module divides the C-end user data into blocks to calculate the dynamic parameters of each data block; according to the dynamic parameters of each data block, the urgency is determined, and the data blocks are sorted and transmitted according to the urgency; The generation module is used to receive data blocks, generate thermal data and mine historical behavior data by tracking user browsing paths, analyzing jump relationships and identifying loss nodes, so as to record the user's operation behavior data on the client; The data analysis module identifies and corrects errors in user data and operation behavior data, performs feature selection and data reduction, and analyzes the reduced data to obtain analyzed data; The extraction module determines the data classification task, extracts the user data information of each client, including basic user information, behavior and transaction data, and classifies them to obtain classified user data; The training module integrates the analyzed data and the classified user data, performs training and reasoning on the integrated data, and obtains patterns and relationships between the data; The intelligent management module predicts user behavior based on patterns and relationships between data to achieve intelligent management of C-end user data.

[0006] Furthermore, starting from the initial URL, by requesting and parsing the target site data, the new URL is obtained after deduplication and filtering, and is put into the queue for cyclic crawling to obtain C-end user data, including: Determine the crawling targets of the C-end user data to be crawled, including corresponding websites, forums or social media platforms; According to the crawling target, obtain the web page link as the starting point of the crawler, that is, the initial URL; Use the HTTP library to initiate a network request to the target site, obtain the web page content, and parse the web page through the parsing library to extract the required data; De-duplicate and filter the required data to extract new URL links; Put the new URL link into a queue, repeat the extraction process, and continuously take out the URL from the queue for crawling until the preset crawling depth is reached to obtain C-end user data.

[0007] Further, the C-end user data is divided into blocks to calculate the dynamic parameters of each data block; the urgency is determined according to the dynamic parameters of each data block, and the data blocks are sorted and transmitted according to the urgency, including: Calculate the average value of the C-end user data to get a center point; calculate the standard deviation based on the C-end user data; Determine a corresponding radius based on the standard deviation to define a circle; Traverse each data point and calculate its Euclidean distance to the center point. If the distance is less than or equal to the radius, the data point is considered as a "point inside the circle". If the distance is greater than the radius, the data point is "projected" onto the circle through a mapping method and becomes a "point on the circle". Select "points inside the circle" and "points on the circle" as the data sets to be processed. According to the distribution and quantity of data, the processed data set is divided into multiple data blocks using an equidistant segmentation algorithm; For each data block, dynamic parameters are calculated; Analyze the dynamic parameters to obtain the urgency index corresponding to each data block; According to the urgency index, all data blocks are sorted from high to low, and the data blocks are transmitted in sequence according to the sorted order.

[0008] Furthermore, the calculation formula of the dynamic parameter is: ; in, is a dynamic parameter, is the weight coefficient of the real-time factor, is the base of natural logarithms, is the decay rate, is the current time, is the creation time of the data block, is the weight coefficient, ranging from 0.1 to 10; is the number of data blocks, It is t The coefficients of the data blocks, It is t The frequency of occurrence of a data block in user behavior, is the data size weight coefficient, is the maximum number of bytes in the data block, is the number of bytes in the data block, is the index.

[0009] Furthermore, the dynamic parameters are analyzed to obtain the urgency index corresponding to each data block, including: Analyze dynamic parameters and determine the range of parameters; According to the range of the parameters, different value ranges of dynamic parameters are mapped to different levels of urgency, and the mapping rules are represented by a decision tree structure, where each node represents a judgment condition of a dynamic parameter, branches represent different value ranges, and leaf nodes represent the results of the urgency; For each data block, extract its dynamic parameters. Starting from the root node, divide the data set into subsets based on the dynamic parameters and create child nodes. Evaluate each node. If the performance improvement after division is not obvious, stop the division and mark the current node as a leaf node. Recursively repeat the above process on each child node until the stopping condition is met, and traverse each non-leaf node from bottom to top to evaluate its performance after being replaced by a leaf node; According to the performance evaluation results after the leaf nodes, each leaf node is marked as a corresponding urgency category to obtain the urgency index corresponding to each data block.

[0010] Furthermore, by tracking user browsing paths, analyzing jump relationships, and identifying lost nodes, we generate thermal data and mine historical behavior data to record user operation behavior data on the client, including: By tracking the user's browsing path in the client, building the user's navigation map, and analyzing the user's jump relationship between different pages, identifying the key nodes of user loss, and obtaining path analysis results; Generate thermal data of the page based on the path analysis results and operation behaviors, and obtain the user's historical behavior data; Mine and analyze users' historical behavior data, track events, and record users' operational behavior data on the client.

[0011] Furthermore, errors in user data and operation behavior data are identified and corrected, feature selection and data reduction are performed, and the reduced data is analyzed to obtain analyzed data, including: Identify erroneous data in user data and operational behavior data, and correct the erroneous data to obtain corrected data; Perform data selection on the corrected data, perform standardization on the selected feature data, and obtain a standardized data matrix; For the standardized data matrix, calculate the covariance matrix, and perform eigenvalue decomposition on the covariance matrix to obtain the eigenvalue; Sort the eigenvalues ​​from large to small and k The eigenvector corresponding to the largest eigenvalue is taken as the principal component, and the principal component matrix is ​​obtained; Project the standardized feature matrix of the original data onto the principal component matrix to obtain a low-dimensional data matrix after dimensionality reduction; Perform transformation analysis on the reduced data to obtain analyzed data.

[0012] Further, determine the data classification task, extract the user data information of each client, including basic user information, behavior and transaction data, and classify them to obtain classified user data, including: Determine the task plan for data classification, divide the tasks into different clients, and extract the user data information of the client, including basic user information, behavior data and transaction data. The basic user information includes user ID, name, gender, age, address and contact information; Set classification standards and rules, classify the user data information data of the client, and obtain the classified user data.

[0013] Furthermore, the analyzed data and classified user data are integrated, and training and reasoning are performed on the integrated data to obtain patterns and relationships between the data, including: Integrate the analyzed data and classified user data to obtain a unified data set; Using the data set, training the deep learning network to obtain a trained deep learning network; New user data is predicted and analyzed through the trained deep learning network to discover potential patterns and relationships in the data.

[0014] Furthermore, user behavior prediction is performed based on the patterns and relationships between data to achieve intelligent management of C-end user data, including: Build user profiles based on patterns and relationships between data; Calculate the similarity between users based on user portraits and historical behavior data; Based on the similarity, for user behavior data with time sequence, time series analysis is performed to predict the user's future behavior trend; Based on the similarity, for discrete user behavior data, analyze the user's historical behavior data and predict the user's future behavior trend through the recommendation of user portraits; Based on the predicted future behavior trends of users, intelligent management of C-end user data can be achieved.

[0015] After adopting the above technical scheme, the present invention has the following beneficial effects compared with the prior art.

[0016] Automated scripts reduce the time and labor costs of manual data collection, improve data collection efficiency, ensure the uniqueness and accuracy of data through deduplication and filtering, avoid repeated processing, use queues for cyclic crawling, achieve continuous and efficient data collection, and ensure the timeliness and comprehensiveness of data. Process C-end user data in blocks, optimize the data processing process, improve processing efficiency, determine the urgency by calculating dynamic parameters, realize data priority, ensure that important data is processed in a timely manner, improve the efficiency and response speed of data transmission, and meet real-time requirements. Track user browsing paths and analyze jump relationships, deeply understand user behavior, accurately identify key nodes of user loss, generate thermal data and mine historical behavior data, and record user operation behavior data on the client. Identify and correct data errors, improve data quality, ensure the accuracy of analysis results, perform feature selection and data reduction, improve analysis efficiency, and deeply analyze the reduced data to reveal potential trends and patterns.

[0017] Determine clear data classification tasks to make data processing more focused and improve processing efficiency. Extract comprehensive user data information, including basic user information, behavior and transaction data, and classify user data to make data more orderly and clear, thereby improving data availability and value. Integrate analyzed data and classified user data to form a more complete data set, improve data utilization, use high-quality data to train deep learning networks, improve model performance and accuracy, and discover deep patterns and associations between data through training and reasoning, providing a strong data foundation for prediction and decision support. Build refined user portraits, predict future user behavior trends through time series analysis and user portrait recommendations, and improve prediction accuracy. Based on the predicted future user behavior trends, formulate intelligent management strategies for C-end users to improve user experience and service quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings: Figure 1 It is a schematic diagram of the C-end user data intelligent management system based on the big model of the present invention. DETAILED DESCRIPTION

[0019] In order to enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work should fall within the scope of protection of the present application.

[0020] The following embodiment of the present application takes the C-end user data intelligent management system based on a large model as an example to explain the solution of the present application in detail, but this embodiment cannot limit the scope of protection of the present application.

[0021] like Figure 1 As shown, the present invention provides a C-end user data intelligent management system based on a large model, including: Data acquisition module 11, starting from the initial URL, obtains the new URL by requesting and parsing the target site data, filtering and removing duplicates, and puts it into the queue for cyclic crawling to obtain C-end user data; The data transmission module 12 divides the C-end user data into blocks to calculate the dynamic parameters of each data block; determines the urgency according to the dynamic parameters of each data block, and sorts and transmits the data blocks according to the urgency; The generating module 13 is used to receive the data block, generate thermal data and mine historical behavior data by tracking the user's browsing path, analyzing the jump relationship and identifying the lost nodes, so as to record the user's operation behavior data on the client; The data analysis module 14 identifies and corrects errors in user data and operation behavior data, performs feature selection and data reduction, and analyzes the reduced data to obtain analyzed data; The extraction module 15 determines the data classification task, extracts the user data information of each client, including the user's basic information, behavior and transaction data, and classifies them to obtain the classified user data; The training module 16 integrates the analyzed data and the classified user data, performs training and reasoning on the integrated data, and obtains patterns and relationships between the data; The intelligent management module 17 predicts user behavior based on patterns and relationships between data to achieve intelligent management of C-end user data.

[0022] In a specific embodiment of the present invention, data is automatically captured from the target site to improve data collection efficiency. Through deduplication and filtering, the uniqueness and accuracy of data are ensured to avoid repeated processing. Queues are used for cyclic crawling to achieve continuous and efficient data collection. C-end user data is processed in blocks to optimize the data processing process. The urgency is determined by calculating dynamic parameters to achieve data priority sorting, ensure that important data is processed in a timely manner, improve the efficiency and response speed of data transmission, track user browsing paths and analyze jump relationships, deeply understand user behavior, reveal user preferences and usage habits by generating thermal data and mining historical behavior data, and record user operation behavior data on the client. Identify and correct data errors, improve data quality, perform feature selection and data reduction, simplify data models, improve analysis efficiency, deeply analyze reduced data, and reveal potential trends and patterns. Determine clear data classification tasks to make data processing more focused, extract comprehensive user data information, including user basic information, behavior and transaction data, lay the foundation for in-depth analysis, and classify user data. Integrate the analyzed data and classified user data to form a more complete data set. Through training and reasoning, discover the deep patterns and associations between data, and provide a strong data foundation for prediction and decision support. Use the patterns and relationships between data to predict user behavior, enhance market insight, realize intelligent management of C-end user data, and improve user experience and service quality.

[0023] In a preferred embodiment of the present invention, starting from the initial URL, by requesting and parsing the target site data, a new URL is obtained after deduplication and filtering, and it is put into a queue for cyclic crawling to obtain C-end user data, including: Determine the crawling targets of the C-end user data to be crawled, including the corresponding websites, forums or social media platforms, including: Determine the type of C-end user data to be crawled, such as user comments, product reviews, user behavior data, etc. Analyze and select one or more suitable websites, forums or social media platforms as data sources, evaluate the data quality, update frequency and ease of crawling of these platforms, conduct preliminary analysis of the selected websites, and understand their web page structure, data layout and existing anti-crawling mechanisms.

[0024] According to the crawling target, obtain the web page link as the starting point of the crawler, that is, the initial URL, which specifically includes: finding the URL of the website area or page that contains the required data, usually the website's homepage, category page or search results page, ensuring that the obtained URL is accessible and does contain the required data, and preparing multiple initial URLs to expand the scope of data crawling.

[0025] Use the HTTP library to initiate a network request to the target site, obtain the web page content, and parse the web page through the parsing library to extract the required data, including: using the HTTP library to set up and send network requests, configuring the request header to simulate browser behavior to avoid being identified as a crawler by the target website, after sending the request, receiving the server's response, and obtaining the HTML content of the web page, using the parsing library to parse HTML, locate and extract the required data, and write selectors based on the website structure to accurately capture the target data.

[0026] De-duplicate and filter the required data to extract new URL links, including: using data structures to store the identifiers of captured data, checking whether new data already exists in the captured data set before each capture, and using preset rules to filter out invalid, duplicate or irrelevant data. Data cleaning can be performed using regular expressions, conditional judgments and other methods.

[0027] Put the new URL link into a queue, repeat the extraction process, and continuously take out URLs from the queue for crawling until the preset crawling depth is reached to obtain C-end user data, specifically including: creating an empty queue to store the URLs to be crawled, adding the initial URL to the queue, and taking out the URL from the queue for crawling: taking out a URL from the queue and crawling data; during the crawling process, adding the newly discovered valid URL to the queue to ensure that the newly added URL meets the crawling conditions and has not been crawled, repeating the above process until the queue is empty or the preset crawling depth is reached, setting appropriate stop conditions, such as crawling depth, crawling time, and amount of crawled data; storing the captured valid data in a database, file or other storage medium.

[0028] In a specific embodiment of the present invention, the time and labor cost of manually collecting data can be reduced through automated scripts, the scope and source of data collection can be clearly defined, making data collection more focused and efficient, and data sources can be selected in a targeted manner to avoid wasting time and resources on unnecessary or low-quality data sources, providing a clear starting point for data crawling and ensuring the controllability of the crawling process; the use of HTTP libraries and parsing libraries can efficiently obtain and parse web page content, improving the speed of data collection; through professional parsing libraries, key information in web pages can be more accurately extracted, duplicate and irrelevant data can be removed, and the quality and accuracy of the collected C-end user data can be ensured; the waste of storage space can be reduced, and only valuable data can be saved. By continuously taking out URLs from the queue for crawling, the data of the target site can be deeply mined to obtain more comprehensive C-end user data. The automated cyclic crawling process reduces the need for manual intervention and improves the efficiency and sustainability of data collection.

[0029] In a preferred embodiment of the present invention, the C-end user data is divided into blocks to calculate the dynamic parameters of each data block; the urgency is determined according to the dynamic parameters of each data block, and the data blocks are sorted and transmitted according to the urgency, including: Calculate the average value of C-end user data to get a center point; calculate the standard deviation based on the C-end user data, specifically including: obtain C-end user data, calculate the average value for each dimension in the data set, these average values ​​will constitute a center point, representing the "average user" of the data set, calculate the standard deviation for each dimension in the data set, the standard deviation is an important indicator to measure the degree of dispersion of data distribution.

[0030] Determine a corresponding radius based on the standard deviation to define a circle, specifically including: based on the standard deviation calculated above, select a suitable multiple (such as 1 times, 2 times the standard deviation, etc.) as the radius. This radius will be used to define a circle. The data points within the circle will be regarded as data points similar or close to the center point. The calculated center point is the center of the circle, and the circle is drawn with the determined radius.

[0031] Traverse each data point and calculate its Euclidean distance with the center point. If the distance is ≤ the radius, the data point is regarded as an "in-circle point". If the distance is greater than the radius, the data point is "projected" onto the circle through a mapping method to become a "point on the circle". Select "in-circle points" and "points on the circle" as the processed data sets, specifically including: traverse each data point in the C-end user data, and for each data point, calculate its Euclidean distance with the center point. If the Euclidean distance between the data point and the center point is less than or equal to the radius, the data point is regarded as an "in-circle point". If the Euclidean distance between the data point and the center point is greater than the radius, the data point is "projected" onto the circle through a mapping method to become a "point on the circle". The "in-circle points" and "points on the circle" are merged to form a new processed data set.

[0032] According to the distribution and quantity of the data, an equidistant segmentation algorithm is used to divide the processed data set into multiple data blocks, specifically including: performing distribution analysis on the data in the processed data set to understand the dense and sparse areas of the data, and according to the distribution and quantity of the data, an equidistant segmentation algorithm is used to divide the processed data set into multiple data blocks. The equidistant segmentation algorithm ensures that each data block contains a similar number of data points and the intervals between data blocks are relatively uniform.

[0033] For each data block, dynamic parameters are calculated; Analyze the dynamic parameters to obtain the urgency index corresponding to each data block; According to the urgency index, all data blocks are sorted from high to low, and the data blocks are transmitted in sequence according to the sorted order.

[0034] In a specific embodiment of the present invention, by calculating the standard deviation and determining a corresponding radius, the process actually performs a standardization process on the data, which can make the data points more concentrated. By mapping the data points to "points inside the circle" and "points on the circle", the influence of extreme values ​​can be removed, making the data set more robust. By dividing the processed data set into multiple data blocks, these data blocks can be processed in parallel, thereby improving the efficiency of data processing. Calculating dynamic parameters for each data block can more flexibly reflect the characteristics and changing trends of the data to adapt to different data distributions and quantities, thereby providing more accurate data analysis results. By analyzing the dynamic parameters, the urgency index corresponding to each data block is obtained. This index can be used as a basis for prioritizing data processing, thereby improving the pertinence and efficiency of data processing. All data blocks are sorted according to the urgency index, and the data blocks are transmitted in sequence in the sorted order, which can ensure that important data blocks can be processed and transmitted first.

[0035] In a preferred embodiment of the present invention, the calculation formula of the dynamic parameter is: ; in, is a dynamic parameter, is the weight coefficient of the real-time factor, is the base of natural logarithms, is the decay rate, is the current time, is the creation time of the data block, is the weight coefficient, ranging from 0.1 to 10; is the number of data blocks, It is t The coefficients of the data blocks, It is tThe frequency of occurrence of a data block in user behavior, is the data size weight coefficient, is the maximum number of bytes in the data block, is the number of bytes in the data block, is the index.

[0036] In a specific embodiment of the present invention, by introducing a weight coefficient of a real-time factor, newly generated data can be better processed. By using the natural logarithm and the decay rate to adjust the importance of the data, the priority of the old data can be gradually reduced. The adjustability of the weight coefficient and the decay rate enables customization according to the specific needs of different application scenarios. By considering the number, size and frequency of occurrence of data blocks in user behavior, processing resources can be allocated more effectively, which helps to avoid processing bottlenecks and ensure that high-frequency or large data blocks receive appropriate attention. By comprehensively considering multiple dynamic parameters, the priority of data processing can be determined more intelligently, thereby improving the overall processing efficiency, which can not only reduce processing delays, but also help to improve throughput and response speed. By considering the data size weight coefficient and the number of bytes of the data block, storage and transmission resources can be allocated more reasonably, which helps to avoid waste of resources and ensure that key data is supported by sufficient resources.

[0037] In a preferred embodiment of the present invention, the dynamic parameters are analyzed to obtain the urgency index corresponding to each data block, including: Analyze the dynamic parameters and determine the range of the parameters, specifically including: collect historical data sets containing dynamic parameters, average the dynamic parameter values ​​in the entire historical data set to obtain the mean of the dynamic parameters; by observing the distribution of the dynamic parameter values ​​in the data set, you can have a preliminary understanding of the variation range and central trend of the parameters; by calculating the maximum value, minimum value, standard deviation and other statistics of the dynamic parameters, you can analyze its variation range; combine the statistical analysis results to determine the reasonable range of the dynamic parameters, for example, you can set the mean plus or minus the standard deviation as the upper and lower limits of the range. Through the above steps, based on the historical data set and the mean method, you can effectively analyze the distribution and variation range of the dynamic parameters, and determine their reasonable range accordingly.

[0038] According to the range of the parameters, different value ranges of dynamic parameters are mapped to different emergency levels, and the mapping rules are represented by the structure of a decision tree, in which each node represents a judgment condition of a dynamic parameter, branches represent different value ranges, and leaf nodes represent the results of the emergency level, including: determining the number of emergency levels to be divided, for example, three levels of low, medium and high; setting level names and identifiers: setting a certain name and identifier for each level, such as "low" corresponding to identifier 1, "medium" corresponding to identifier 2, and "high" corresponding to identifier 3; performing statistical analysis on the collected historical data of dynamic parameters, including minimum value, maximum value, average value, standard deviation, etc.; based on the results of statistical analysis, the dynamic The parameter value range is divided into several intervals, each interval corresponds to an urgency level; determine the urgency level corresponding to each dynamic parameter value range, for example, parameter values ​​in the range of 0-30 correspond to "low", 31-70 correspond to "medium", and 71-100 correspond to "high"; record the mapping rules in detail in the document, including the name of the dynamic parameter, the value range, and the corresponding urgency level; select a key dynamic parameter as the root node of the decision tree, which should have a significant impact on the urgency; according to the mapping rules, add branches for each dynamic parameter value range, and create leaf nodes at the end of the branches; mark each leaf node as the result of the corresponding urgency level.

[0039] For each data block, extract its dynamic parameters, start from the root node, divide the data set into subsets according to the dynamic parameters, create child nodes, evaluate each node, if the performance improvement after the division is not obvious, stop the division, and mark the current node as a leaf node, specifically including: for each data block, extract its dynamic parameter value, divide the data set into different subsets according to the judgment condition of the root node, create a child node for each divided subset, evaluate the performance after the division, if the performance improvement is not obvious, stop the division, and mark the current node as a leaf node. Among them, the specific calculation formula for evaluating the performance after the division is: ; in, It is a subset D i Middle The probability of class samples; m is the total number of classes; It is a subset D i The weight of represents the proportion of its sample number to the total sample number of the data set; Represents the index of the subset obtained after the dataset D is divided by a certain feature A; Representative subset D i Category index in ; Represents the total number of subsets into which feature A divides dataset D; represents the weighted Gini impurity.

[0040] The above process is repeated recursively on each child node until the stopping condition is met, and each non-leaf node is traversed from bottom to top to evaluate its performance after being replaced by a leaf node, including: for each newly created child node, the above process is repeated, that is, the data set is continued to be divided according to dynamic parameters and deeper child nodes are created; the stopping condition of the recursive division is set, such as reaching a predetermined tree depth; when the stopping condition is met, the recursive process is terminated.

[0041] According to the performance evaluation results after the leaf nodes, each leaf node is marked with the corresponding urgency category to obtain the urgency index corresponding to each data block, specifically including: starting from the bottom of the decision tree, traversing each non-leaf node upward, and evaluating the performance after replacing the current non-leaf node with a leaf node (such as using cross-validation and other methods); if the performance is improved after replacement or meets other optimization criteria, the non-leaf node is replaced with a leaf node, and the leaf node is marked as the most common urgency category in the current subset, ensuring that all leaf nodes are marked with the corresponding urgency category.

[0042] In a specific embodiment of the present invention, through the structure of the decision tree, the relationship between different dynamic parameter values ​​and the urgency can be intuitively seen, the judgment process of the urgency is automated, the need for manual intervention is reduced, and thus the processing efficiency is improved. Since the construction of the decision tree is based on the range and value of the dynamic parameters, when the distribution or range of the dynamic parameters changes, the decision tree can be adjusted accordingly to adapt to the new data situation. By recursively constructing and evaluating the decision tree, it can be ensured that better performance is achieved when dividing the data blocks, which helps to more accurately judge the urgency of the data blocks and avoid misjudgment or missed judgment. According to the urgency index of the data block, processing resources can be allocated more reasonably. The structure of the decision tree is easy to expand and maintain. When it is necessary to introduce new dynamic parameters or adjust existing parameters, the decision tree can be easily updated without large-scale changes to the entire. The urgency index can provide valuable information for decision makers to help them respond faster.

[0043] In a preferred embodiment of the present invention, a data block is received, and thermal data and historical behavior data are generated by tracking the user's browsing path, analyzing the jump relationship and identifying the loss node, so as to record the user's operation behavior data on the client, including: By tracking the user's browsing path in the client, building the user's navigation map, and analyzing the jump relationship between users in different pages, the key nodes of user loss are identified to obtain the path analysis results, which specifically include: using front-end embedding technology, such as JavaScript tags or SDK, to capture the user's click, browsing, scrolling and other behavioral data in the client, ensuring that data collection complies with the privacy policy and obtains user consent; using the collected data to draw the user's jump path between different pages, and use graph databases or visualization tools to present the user's navigation path to form a map; analyzing the user's flow between various pages, identifying which pages are frequently visited by users and which jump paths are frequently taken by users; using data analysis tools, such as pivot tables or data analysis software, to count and analyze jump data; by analyzing indicators such as the user's stay time on different pages and the bounce rate, identify the key nodes of user loss, and use tools such as funnel models to analyze at which links the user is lost the most, so as to locate the problem.

[0044] Based on the path analysis results and operation behaviors, the thermal data of the page is generated to obtain the user's historical behavior data, including: through the front-end tracking technology, the user's complete access path on the page is captured and recorded, including the order of pages visited, the dwell time, etc.; at the same time, the user's specific operation behaviors on each page, such as clicks, scrolling, hovering, etc. are collected; the collected path data and operation behavior data are integrated to ensure the relevance and time series between the data, and the user's operation behavior is mapped to specific page elements or areas; based on the user's click frequency, scrolling depth, dwell time and other operation behavior indicators in each area of ​​the page, the heat value of each area is calculated. The heat value can be a comprehensive indicator that reflects the user's attention and interactive activity in the area; the calculation formula of the heat value of each area includes: Counting the number of clicks by users in each area can be used directly as part of the heat value of the area; accumulating the time each user stays in each area and then calculating the average to reflect the average length of time the user pays attention to the area, which can be used as a weighting factor for the heat value; judging the user's attention to the area based on the scrolling distance or scrolling time in the area, the scrolling depth can be converted into a specific value and added to the calculation of the heat value; taking a weighted sum of indicators such as the number of clicks, dwelling time, and scrolling depth to obtain a comprehensive heat value for each area.

[0045] Mining and analyzing the user's historical behavior data, tracking events, and recording the user's operational behavior data on the client side, specifically including: continuously collecting and saving all the user's historical behavior data through client log recording, database storage, etc., including page browsing records, button click events, form filling behaviors, etc.; cleaning the collected historical behavior data, removing invalid, duplicate or erroneous data records, and performing data preprocessing, such as formatting and standardization, to ensure data quality and consistency; tracking events for the user's key operational behaviors, such as purchase, registration, adding to shopping cart, etc., assigning a unique identifier to each event, and recording the specific time of the event, user ID and other related information; using data mining techniques, such as association rule mining, sequence pattern mining, etc., to analyze the correlation and sequential patterns between user behaviors, discover user preferences, habits and potential needs, provide data support for personalized recommendations and marketing strategies, and generate user operational behavior data reports based on mining and analysis results.

[0046] In a specific embodiment of the present invention, by constructing a user's navigation map, the user's browsing path and jump relationship in the client can be intuitively understood. By analyzing the jump relationship between different pages of the user, the key nodes that cause user loss can be accurately identified, providing a strong basis for optimizing user experience and reducing user loss. The path analysis results can provide data support for product optimization, page design, content recommendation, etc., helping to improve user satisfaction and retention rate. By generating thermal data of the page, the user's click, browse and other operation behaviors on the page can be intuitively displayed, which is helpful to discover the hot spots of the page and the focus of user attention. Combining the path analysis results and operation behaviors, more comprehensive user historical behavior data can be obtained, and thermal data can provide feedback for page design. By mining and analyzing the user's historical behavior data, the rules and trends of user behavior can be revealed, and event tracking and recording of the user's operation behavior data on the client can ensure the integrity and accuracy of the data.

[0047] In a preferred embodiment of the present invention, identifying and correcting errors in user data and operation behavior data, performing feature selection and data reduction, and analyzing the reduced data to obtain analyzed data include: Identify erroneous data in user data and operational behavior data, and correct the erroneous data to obtain corrected data, specifically including: reviewing user data and operational behavior data one by one, checking whether there are missing values, outliers, duplicate values ​​or format errors, and using statistical methods (such as box plots, IQR rules, etc.) to identify outliers; for missing values, select a filling method based on the distribution of the data, such as mean filling, median filling or using interpolation methods; for outliers, decide whether to delete or replace with reasonable values ​​based on data characteristics, correct format errors, and ensure that all data meets the expected format standards.

[0048] The corrected data is subjected to data selection, and the selected characteristic data is subjected to standardization processing to obtain a standardized data matrix, which specifically includes: Analyze user data and operational behavior data, select features that are highly relevant to business goals, and use feature selection techniques (such as variance analysis, correlation analysis, etc.) to remove redundant or irrelevant features; standardize the selected features, that is, scale each feature value so that its mean is 0 and its standard deviation is 1.

[0049] For the standardized data matrix, the covariance matrix is ​​calculated, and the eigenvalue decomposition of the covariance matrix is ​​performed to obtain the eigenvalues; the eigenvalues ​​are sorted from large to small and k The eigenvector corresponding to the largest eigenvalue is used as the principal component to obtain the principal component matrix, which specifically includes: determining the standardized data matrix, in which each column represents a feature and each row represents a sample. The standardized data ensures that the mean of each feature is 0 and the standard deviation is; for each pair of features (columns) in the standardized data matrix, calculate the covariance between them; fill the covariance values ​​between all the features in pairs into the matrix to form a square matrix. The diagonal elements of the matrix are the covariance of each feature with itself, that is, the variance, which is 1 in the standardized data (because the standard deviation is 1); use the eigenvalue decomposition method in linear algebra (such as QR algorithm, Jacobi iteration, etc.) to decompose the covariance matrix Solution, after decomposition, two groups of outputs are obtained: one group is the eigenvalues ​​(scalars), which represent the "importance" of the eigenvectors corresponding to the covariance matrix; the other group is the corresponding eigenvectors (vectors), which represent the main direction of data change. The eigenvalues ​​are sorted from large to small, and the size of the eigenvalue reflects the amount of data variability represented by the corresponding eigenvector; larger eigenvalues ​​mean that the corresponding eigenvector explains more data changes; select the first k largest eigenvalues ​​and their corresponding eigenvectors as needed. These eigenvectors constitute the "principal components" of principal component analysis. The selection of k can be based on the cumulative contribution rate of the eigenvalues ​​(that is, the proportion of the sum of the first k eigenvalues ​​to the total sum of eigenvalues) or a preset threshold.

[0050] The standardized feature matrix of the original data is projected onto the principal component matrix to obtain a low-dimensional data matrix after dimensionality reduction, specifically including: selecting the eigenvectors corresponding to the first k largest eigenvalues ​​through eigenvalue decomposition, and organizing these eigenvectors into a principal component matrix by column, denoted as W; using matrix multiplication to project the standardized feature matrix X onto the principal component matrix W, the projection formula is: Y = XW, where Y is the low-dimensional data matrix after dimensionality reduction, each row in Y still corresponds to a sample, but the number of columns is now reduced to k, that is, the number of selected principal components. In this way, the original high-dimensional data is projected into a low-dimensional space while retaining the most important change direction and information.

[0051] The reduced data are transformed and analyzed to obtain the analyzed data, specifically including: applying the selected transformation method to each element in the reduced low-dimensional data matrix, for example, if logarithmic transformation is selected, taking the natural logarithm of each non-zero element in the data matrix; performing descriptive statistical analysis on the transformed data, including calculating the mean, median, standard deviation, quartiles, etc.; understanding the central tendency of the data through the mean and median, understanding the degree of dispersion of the data through the standard deviation, and understanding the distribution pattern of the data and possible outliers through the quartiles.

[0052] In a specific embodiment of the present invention, the data quality is significantly improved by identifying and correcting errors in user data and operational behavior data. The data is selected and the selected features are standardized, which helps to eliminate the dimensional differences between different features. The dimension of the data is reduced by principal component analysis, which not only reduces the computational complexity and improves the processing speed, but also helps to eliminate redundant information in the data, making the analysis results clearer and more intuitive. The principal components extracted by PCA often represent the main change trends and key information in the data. The analysis efficiency and accuracy of the data processed by the above process will be significantly improved; conversion analysis of the reduced data provides more flexibility and possibilities.

[0053] In a preferred embodiment of the present invention, a data classification task is determined, user data information of each client is extracted, including basic user information, behavior and transaction data, and classified to obtain classified user data, including: Determine the purpose of data classification, such as customer segmentation, behavior analysis, risk assessment, etc.; decompose the overall task into several subtasks, such as data collection, data preprocessing, feature extraction, classification model selection and evaluation, etc., set time nodes for each subtask, and develop a project schedule; determine the different clients that need to be processed, such as different business departments, product lines or geographical regions, to ensure that the data of different clients can be accessed and processed independently; identify the data source of each client, including databases, APIs, CSV files, etc.; use ETL (Extract, Transform, Load) tools or SQL queries to extract data from the data source; extract the user's basic information, which includes user ID, name, gender, age, address and contact information; handle missing values, duplicate values ​​and outliers to ensure data quality. Extract user behavior data, such as number of logins, page views, click-through rate, search history, etc., analyze the time series characteristics of behavior data, and understand the patterns and trends of user activities. Extract user transaction data, such as purchase records, transaction amounts, payment methods, transaction times, etc., and calculate indicators such as user transaction totals, average transaction amounts, and transaction frequency.

[0054] Select features related to the classification target, such as user value, activity, risk level, etc.; create new features, such as user lifetime value (CLV), last transaction time (Recency), etc.; define classification rules according to business needs, such as dividing users into high-value users, ordinary users and low-value users, and set classification thresholds. For example, users with an annual transaction volume of more than 10,000 yuan are defined as high-value users.

[0055] Analyze the various features in the data set to determine which features are closely related to the classification target. For example, for user classification, the features that may be relevant include age, gender, purchase history, transaction amount, etc. Define the classification target, such as classifying users into high-value users, ordinary users, and low-value users, and convert the classification target into a numerical form so that the model can process it, for example, express the user value level as a number.

[0056] Based on data characteristics and classification objectives, select a decision tree model, divide the data set into training and test sets, use 80% of the data for training and 20% for testing to ensure the independence of model evaluation; set a random seed to ensure that the results of data partitioning and model training are repeatable, instantiate the decision tree classification model in the selected tool, train the model using the training data set, and adjust the model parameters to fit the data.

[0057] Through methods such as grid search or random search, adjust the hyperparameters of the decision tree (such as maximum depth, minimum number of sample splits) to optimize model performance. During the parameter adjustment process, use cross-validation to evaluate the performance of the model under different parameter combinations and select the parameter settings with the best performance.

[0058] Apply the trained decision tree model to a new user data set, make classification predictions for each user, explain the classification results output by the model, use the cross-validation method to evaluate the performance of the model on the training data set, and calculate evaluation indicators such as accuracy, recall, and F1 score; analyze the results of different evaluation indicators to identify possible weaknesses and room for improvement of the model; conduct A / B testing in an actual business environment, apply the classification results of the model to actual users, compare the impact of different classification strategies on business indicators, and evaluate the actual effect of the classification model based on business indicators (such as conversion rate, user satisfaction); select a suitable database or data warehouse (such as MySQL, PostgreSQL, Amazon Redshift) to store the classification results based on the data volume and access requirements, design the corresponding data table structure, including fields such as user ID, classification label, classification time, etc., write the classification results to the selected database or data warehouse, create indexes for commonly used query fields, and optimize data retrieval performance.

[0059] In a specific embodiment of the present invention, by determining a task plan for data classification, the goal and scope of classification can be clarified to ensure the orderly progress of classification work. Dividing tasks into different clients for processing can make full use of resources, realize division of labor and cooperation, improve the efficiency of data processing, and extract user data information data of the client, including basic user information, behavior data and transaction data, to ensure the comprehensiveness and accuracy of the data. By setting clear classification standards and rules, the consistency and accuracy of data classification can be ensured, and the subjectivity and arbitrariness in the classification process can be avoided. The classified user data is more orderly and clear, which improves the availability and value of the data. The classified user data can provide strong support for precision marketing.

[0060] In a preferred embodiment of the present invention, the analyzed data and the classified user data are integrated, and the integrated data is trained and inferred to obtain patterns and relationships between the data, including: Integrate the analyzed data and classified user data to obtain a unified data set, including: merging data sets from different sources according to specific keys (such as user ID) to ensure that each user's data entry is complete; splicing data with different characteristics together to form a complete data entry containing all relevant information; for missing data, use methods such as mean filling, median filling, most frequent value filling or interpolation to process; data set division: Training set: The subset of data used for model training, usually accounting for 60-80% of the entire dataset.

[0061] Validation set: A data subset used for model tuning and selection, usually accounting for 10-20%.

[0062] Test set: A subset of data used for model evaluation, usually 10-20%.

[0063] Using the data set, the deep learning network is trained to obtain the trained deep learning network, specifically including: selecting a suitable deep learning architecture (such as convolutional neural network CNN) according to the data characteristics and task requirements; determining the number of network layers, the number of neurons in each layer, the activation function, the optimizer and other hyperparameters; standardizing, normalizing or other necessary preprocessing operations on the training set data; inputting the preprocessed training set data into the selected deep learning network, and calculating the output through forward propagation; using the backpropagation algorithm to calculate the gradient of the loss function, and updating the network weights and biases through the gradient descent algorithm, repeating the above process, and iterating the network for multiple times until the preset stopping condition is reached (such as reaching the maximum number of iterations); evaluating the performance of the model on the validation set, adjusting the hyperparameters to optimize the model, and selecting the model with the best performance based on the performance of the validation set. After training and verification, a trained deep learning network is obtained, which can capture the complex patterns and relationships between data.

[0064] Deep learning networks can identify correlations between different features. For example, in e-commerce data, there may be a correlation between purchase history and the user's age and gender, and the network can learn how these features jointly affect the user's purchasing behavior. For time series data or data with a sequential nature, deep learning networks (such as recurrent neural networks RNN or its variants LSTM, GRU) can capture the pattern of data changes over time and how data at different time points affect each other. In image or video data, deep learning networks (such as convolutional neural networks CNN) can learn the spatial structural relationship between pixels or frames to identify objects, scenes or actions. Through a multi-layer structure, deep learning networks can extract and represent hierarchical features of data layer by layer. For example, in image recognition, the bottom-level network may learn low-level features such as edges and textures, while the high-level network may learn high-level features of parts or the whole of an object.

[0065] The trained deep learning network is used to predict and analyze new user data to mine potential patterns and relationships in the data, including: using the validation data set to validate the trained deep learning network. The validation process includes calculating the accuracy, recall, F1 score and other indicators on the validation set to evaluate the performance, and using the validated deep learning network to predict and analyze new user data, involving the prediction of user behavior, transaction records and other data, or the analysis of user portraits, user preferences and other data. Through prediction and analysis, the potential patterns and relationships in the data are mined, and the prediction and analysis results are integrated to form a large processing result.

[0066] In a specific embodiment of the present invention, a unified data set containing richer and more comprehensive information is obtained by integrating the analyzed data and the classified user data. Preprocessing the data, such as cleaning, denoising, normalization, etc., can improve the quality of the data, reduce outliers and noise in the data, and make the data more accurate and reliable. The integration and preprocessing process ensures the consistency of the data, so that subsequent data analysis and training can be carried out on the basis of unified data, avoiding the problems caused by inconsistent data. Using high-quality data after preprocessing to train a deep learning network can better learn the features and laws in the data, improve performance and accuracy. The deep learning network has a powerful automatic feature extraction capability, which can automatically extract useful features from complex data, reducing the workload of manual feature engineering. By training a deep learning network, it is possible to learn the general laws in the data, so that it has better generalization capabilities for new data and can adapt to data analysis and prediction needs in different scenarios.

[0067] In a preferred embodiment of the present invention, user behavior prediction is performed based on patterns and relationships between data to achieve intelligent management of C-end user data, including: According to the patterns and relationships between data, user portraits are constructed, including: extracting data related to users and integrating these data into a comprehensive user portrait. Each user portrait contains multiple features, such as age, gender, geographic location, browsing history, etc. According to the characteristics of the data, select features that have an important impact on user behavior prediction, pre-process the features, such as standardization and normalization, to ensure comparability and consistency between features, and combine the processed feature data into user portraits. Each user corresponds to a unique portrait.

[0068] Based on user portraits and historical behavior data, Calculate the similarity between users, where is the total number of features in the user profile, Is a user In the The value of the feature, Is a user In the The value of the feature, It is The weight of the feature, It is The average value of features among all users, It is The standard deviation of a feature among all users, is the similarity, and It is a user portrait. is the index; Based on similarity, for user behavior data with time sequence, time series analysis is used to predict the user's future behavior trend; based on similarity, for discrete user behavior data, historical user behavior data is analyzed, and user portraits are recommended to predict the user's future behavior trend; based on the predicted user's future behavior trend, intelligent management of C-end user data is achieved, including: Extract time-related features, such as time intervals, periodicity, trends, etc. For discrete behavioral features, extract the frequency, type, sequence and other features of user behavior; generate user portraits based on the user's historical behavior data, including interest preferences, consumption capacity, activity, etc.

[0069] Collect users' discrete behavior data, such as clicks, comments, shares, etc., remove invalid data, and encode discrete behaviors into numerical form; build user portraits based on users' discrete behavior data, including interest preferences, consumption habits, activity, etc.; select appropriate ARIMA model parameters (p, d, q) according to the characteristics of time series data (such as trends, seasonality), where p represents the number of autoregressive terms; d represents the number of differences, which is used to make the time series smooth; q represents the number of moving average terms; use preprocessed time series data to train the ARIMA model, use historical data as the training set, estimate model parameters through maximum likelihood estimation and other methods, and use cross-validation methods to verify the model's predictive performance; use the trained ARIMA model to predict users' future behavior trends, such as future purchases and page views.

[0070] Based on user portraits and discrete behavior data, the cosine similarity between users is calculated; user portraits are descriptions of the image and preferences of real users in the system, usually including user demographic characteristics (such as gender, age, region, etc.), behavioral characteristics (such as browsing, clicking, purchase records, etc.) and content characteristics (such as user favorite categories, topics, keywords, etc.). The process of building user portraits usually involves data collection, cleaning, feature extraction and labeling.

[0071] Collecting discrete behavioral data: Discrete behavior data refers to the specific behavior records generated during the interaction between users and the system, such as users' clicks, purchases, and comments on products. These data usually exist in the form of logs and are the basis for building recommendation models.

[0072] Constructing a user-item behavior matrix: Based on the user's discrete behavior data, construct a user-item matrix, where the elements in the matrix represent the user's preference for the item (such as rating, number of clicks, etc.); based on the calculated user similarity, select the K users with the highest similarity to the target user as the similar user set; for item i that the target user u has never heard of, calculate the target user u's recommendation score for item i based on the preference of users in the similar user set S(u, K) for item i, where the target user u's recommendation score for item i is calculated as: ; in, Is a user For items The predicted score of right level of interest; Is with the user Most similar The set of users, i.e., the selection and The one with the greatest similarity users, and use their ratings to infer Ratings; Is a user and users The cosine similarity between the two is used to measure the user and The similarity in interests, the larger the value, the more similar the two users are; Is a user The average rating of all items. By subtracting the mean, we can eliminate the differences in user rating habits. For example, some users tend to give higher ratings, while others tend to give lower ratings. Is a user The set of items that have been rated, in order to predict the user For items The rating needs to be Rated items to help predict; Is a user For items The actual rating of Already on the item If a rating is given, use that rating directly; Is a user and The absolute value of the similarity between them is used for normalization to ensure that the weight of each similarity contribution is within a reasonable range.

[0073] Utilize the trained collaborative filtering model and combine it with user portrait information to predict content or products that users may be interested in. Integrate the results of time series prediction and collaborative filtering recommendation. For example, adjust the weight or priority of collaborative filtering recommendation based on the future behavior trend of users predicted by time series. Provide users with personalized content or product recommendations based on the integrated results. Evaluate the recommendation effect through indicators such as user feedback, click-through rate, and conversion rate, and optimize and adjust the model based on the evaluation results. Update the data set regularly and retrain the ARIMA model and collaborative filtering model as new data is continuously generated. Adjust model parameters or select new models based on changes in business needs and user behavior to improve the accuracy of prediction and recommendation. Through the above steps, it is possible to predict the future behavior trend of users and recommend content or products that users may be interested in, thereby improving user experience and business results.

[0074] In a specific embodiment of the present invention, by constructing a refined user portrait, the preferences, needs and behavior patterns of each user can be understood more accurately. The user portrait helps enterprises to accurately locate the target user group, optimize marketing strategies, and improve the response rate and conversion rate of marketing activities. By calculating the similarity between users, user groups with similar interests or behaviors can be found. Similarity calculation is the basis for building recommendations, which helps to improve the accuracy of recommendations and user satisfaction. By using time series analysis, the trend of user behavior changes over time can be captured, and the user's future behavior can be predicted more accurately, so as to make more accurate decisions. By predicting user behavior, enterprises can provide appropriate services or products when users may need them, thereby enhancing user stickiness and loyalty. Intelligent management of user data can help enterprises automate and optimize many operational processes, reduce manual intervention, and improve work efficiency.

[0075] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A C-end user data intelligent management system based on a large model, characterized in that: include: The data acquisition module starts from the initial URL, requests and parses the target site data, obtains the new URL after deduplication and filtering, and puts it into the queue for cyclic crawling to obtain C-end user data; The data transmission module divides the C-end user data into blocks to calculate the dynamic parameters of each data block; According to the dynamic parameters of each data block, the urgency is determined, and the data blocks are sorted and transmitted according to the urgency; The generation module is used to receive data blocks, generate thermal data and mine historical behavior data by tracking user browsing paths, analyzing jump relationships and identifying loss nodes, so as to record the user's operation behavior data on the client; The data analysis module identifies and corrects errors in user data and operation behavior data, performs feature selection and data reduction, and analyzes the reduced data to obtain analyzed data; The extraction module determines the data classification task, extracts the user data information of each client, including basic user information, behavior and transaction data, and classifies them to obtain classified user data; The training module integrates the analyzed data and the classified user data, performs training and reasoning on the integrated data, and obtains patterns and relationships between the data; The intelligent management module predicts user behavior based on patterns and relationships between data to achieve intelligent management of C-end user data.

2. The C-end user data intelligent management system based on a large model according to claim 1 is characterized in that: Starting from the initial URL, by requesting and parsing the target site data, the new URL is obtained after deduplication and filtering, and is put into the queue for cyclic crawling to obtain C-end user data, including: Determine the crawling targets of the C-end user data to be crawled, including corresponding websites, forums or social media platforms; According to the crawling target, obtain the web page link as the starting point of the crawler, that is, the initial URL; Use the HTTP library to initiate a network request to the target site, obtain the web page content, and parse the web page through the parsing library to extract the required data; De-duplicate and filter the required data to extract new URL links; Put the new URL link into a queue, repeat the extraction process, and continuously take out the URL from the queue for crawling until the preset crawling depth is reached to obtain C-end user data.

3. The C-end user data intelligent management system based on a large model according to claim 2 is characterized in that: Divide the C-end user data into blocks to calculate the dynamic parameters of each data block; Based on the dynamic parameters of each data block, the urgency is determined, and the data blocks are sorted and transmitted according to the urgency, including: Calculate the average value of the C-end user data to get a center point; calculate the standard deviation based on the C-end user data; Determine a corresponding radius based on the standard deviation to define a circle; Traverse each data point and calculate its Euclidean distance to the center point. If the distance is less than or equal to the radius, the data point is considered as a "point inside the circle". If the distance is greater than the radius, the data point is "projected" onto the circle through a mapping method to become a "point on the circle". Select "points inside the circle" and "points on the circle" as the data sets to be processed. According to the distribution and quantity of data, the processed data set is divided into multiple data blocks using an equidistant segmentation algorithm; For each data block, dynamic parameters are calculated; Analyze the dynamic parameters to obtain the urgency index corresponding to each data block; According to the urgency index, all data blocks are sorted from high to low, and the data blocks are transmitted in sequence according to the sorted order.

4. The C-end user data intelligent management system based on a large model according to claim 3 is characterized in that: The calculation formula of dynamic parameters is: ； in, is a dynamic parameter, is the weight coefficient of the real-time factor, is the base of natural logarithms, is the decay rate, is the current time, is the creation time of the data block, is the weight coefficient; is the number of data blocks, It is t The coefficients of the data blocks, It is t The frequency of occurrence of a data block in user behavior, is the data size weight coefficient, is the maximum number of bytes in the data block, is the number of bytes in the data block, is the index.

5. The C-end user data intelligent management system based on a large model according to claim 4 is characterized in that: Analyze the dynamic parameters to obtain the urgency index corresponding to each data block, including: Analyze dynamic parameters and determine the range of parameters; According to the range of the parameters, different value ranges of dynamic parameters are mapped to different levels of urgency, and the mapping rules are represented by a decision tree structure, where each node represents a judgment condition of a dynamic parameter, branches represent different value ranges, and leaf nodes represent the results of the urgency; For each data block, extract its dynamic parameters. Starting from the root node, divide the data set into subsets based on the dynamic parameters and create child nodes. Evaluate each node. If the performance improvement after division is not obvious, stop the division and mark the current node as a leaf node. Recursively repeat the above process on each child node until the stopping condition is met, and traverse each non-leaf node from bottom to top to evaluate its performance after being replaced by a leaf node; According to the performance evaluation results after the leaf nodes, each leaf node is marked as a corresponding urgency category to obtain the urgency index corresponding to each data block.

6. The C-end user data intelligent management system based on a large model according to claim 5 is characterized in that: By tracking user browsing paths, analyzing jump relationships, and identifying loss nodes, we generate thermal data and mine historical behavior data to record user operation behavior data on the client, including: By tracking the user's browsing path in the client, building the user's navigation map, and analyzing the user's jump relationship between different pages, identifying the key nodes of user loss, and obtaining path analysis results; Generate thermal data of the page based on the path analysis results and operation behaviors, and obtain the user's historical behavior data; Mine and analyze users' historical behavior data, track events, and record users' operational behavior data on the client.

7. The C-end user data intelligent management system based on a large model according to claim 6 is characterized in that: Identify and correct errors in user data and operational behavior data, perform feature selection and data reduction, and analyze the reduced data to obtain analyzed data, including: Identify erroneous data in user data and operational behavior data, and correct the erroneous data to obtain corrected data; Perform data selection on the corrected data, perform standardization on the selected feature data, and obtain a standardized data matrix; For the standardized data matrix, calculate the covariance matrix, and perform eigenvalue decomposition on the covariance matrix to obtain the eigenvalue; Sort the eigenvalues ​​from large to small and k The eigenvector corresponding to the largest eigenvalue is taken as the principal component, and the principal component matrix is ​​obtained; Project the standardized feature matrix of the original data onto the principal component matrix to obtain a low-dimensional data matrix after dimensionality reduction; Perform transformation analysis on the reduced data to obtain analyzed data.

8. The C-end user data intelligent management system based on a large model according to claim 7 is characterized in that: Determine the data classification task, extract the user data information of each client, including basic user information, behavior and transaction data, and classify them to obtain classified user data, including: Determine the task plan for data classification, divide the tasks into different clients, and extract the user data information of the client, including basic user information, behavior data and transaction data. The basic user information includes user ID, name, gender, age, address and contact information; Set classification standards and rules, classify the user data information data of the client, and obtain the classified user data.

9. The C-end user data intelligent management system based on a large model according to claim 8 is characterized in that: Integrate the analyzed data and classified user data, perform training and reasoning on the integrated data, and obtain patterns and relationships between the data, including: Integrate the analyzed data and classified user data to obtain a unified data set; Using the data set, training the deep learning network to obtain a trained deep learning network; New user data is predicted and analyzed through the trained deep learning network to discover potential patterns and relationships in the data.

10. The C-end user data intelligent management system based on a large model according to claim 9 is characterized in that: Based on the patterns and relationships between data, user behavior prediction is performed to achieve intelligent management of C-end user data, including: Build user profiles based on patterns and relationships between data; Calculate the similarity between users based on user portraits and historical behavior data; Based on the similarity, for the user behavior data with time sequence, time series analysis is performed to predict the user's future behavior trend; Based on the similarity, for discrete user behavior data, analyze the user's historical behavior data and predict the user's future behavior trend through the recommendation of user portraits; Based on the predicted future behavior trends of users, intelligent management of C-end user data can be achieved.

Images