Data analysis and display system based on large model

Abstract

The invention discloses a data analysis and display system based on a large model, and the system comprises a data collection module which is used for carrying out the matching with a business database according to a current business stage, obtaining a matching result, starting a data burying point according to the matching result, and carrying out the data collection, and obtaining initial data; the data processing module is used for deploying a plurality of edge computing nodes on an edge server and performing multi-thread preprocessing on the initial data by utilizing the edge computing nodes; the data analysis module is used for transmitting the preprocessed initial data into an analysis model of the current service stage to obtain an analysis result; and the result display module is used for carrying out classified statistics on the preprocessed initial data and the analysis result to obtain a plurality of data reports and displaying the data reports in a monitoring center. The invention relates to the technical field of e-commerce data analysis, and solves the technical problems of low data processing efficiency and insufficient accuracy and breadth of data analysis in an existing e-commerce data analysis system.

Description

Technical Field

[0001] The present invention belongs to the field of e-commerce data analysis, and relates to data embedding and deep learning technology, and specifically is a data analysis and display system based on a large model. Background Art

[0002] E-commerce data analysis systems can help companies optimize their business operations, monitor sales data in real time, optimize platform page layouts based on user behavior data, and improve user experience. They can help make accurate marketing decisions, segment user groups, develop personalized marketing strategies, and reasonably allocate marketing resources. They can also gain insights into market trends, monitor market dynamics, grasp the overall data trends of the industry, and plan business in advance. At present, the data sources of e-commerce data analysis systems are extensive, covering various types of internal data of enterprises and external data such as social media, logistics, and market research. The application of technology is also becoming more diversified, involving big data, machine learning, and deep learning technologies. In addition, the analysis needs of enterprises are becoming increasingly complex, and they expect to deeply explore the value of data from multiple dimensions.

[0003] However, in terms of data processing and data analysis, e-commerce data analysis systems still have many shortcomings. In data processing, data integration faces huge challenges. The formats of different data sources vary greatly, data quality problems occur frequently, and timeliness is difficult to guarantee. At the same time, the use of computing resources is unreasonable, and traditional methods are difficult to cope with large-scale data processing needs. In terms of data analysis, the model has limitations, poor versatility, and insufficient understanding of complex data relationships. The prediction accuracy is not high, and it is difficult to accurately grasp the causal relationship between data, which limits the accuracy and foresight of corporate decision-making. Summary of the invention

[0004] The present invention aims to solve at least one of the technical problems existing in the prior art; to this end, the present invention proposes a data analysis and display system based on a large model, which is used to solve the technical problems of low data processing efficiency and insufficient accuracy and breadth of data analysis in existing e-commerce data analysis systems.

[0005] To achieve the above objectives, the present invention provides a data analysis and display system based on a large model, comprising:

[0006] Data collection module: used to match the current business stage with the business database to obtain matching results, and start data collection based on the matching results to obtain some initial data; the matching results indicate the current business stage and the data collection ratio, and the data collection ratio indicates the ratio of the number of collected data items to the data items included in each data category;

[0007] Data processing module: used to deploy several edge computing nodes on several edge servers, and use several edge computing nodes to perform multi-threaded preprocessing on several initial data to obtain several preprocessed initial data;

[0008] Data analysis module: used to transmit the pre-processed initial data to the monitoring center, call the analysis model of the corresponding stage according to the current business stage, and pass the pre-processed initial data to the analysis model of the corresponding stage to obtain the analysis result; wherein, the analysis model of the corresponding stage is obtained after fine-tuning based on the pre-trained general model, and the pre-trained general model is constructed based on deep learning;

[0009] Result display module: used to classify and count the initial data and analysis results after preprocessing, obtain several data reports and display them in the monitoring center.

[0010] Based on the above technical modules, an efficient, intelligent and comprehensive e-commerce data analysis and decision support system has been implemented. Among them, the data collection module collects initial data according to the current business stage to ensure accurate acquisition of data items related to the current business stage and the appropriate number of data items, avoiding the blindness of data collection, so that the collected data can better reflect the business characteristics and needs of each stage; through the data processing module, distributed data preprocessing is performed using edge computing nodes and multi-threaded processing technology to improve the efficiency of data processing; through the data analysis module, large model technology is used to mine complex relationships in the data and accurately analyze data at different business stages; through the result display module, data and analysis results are classified, counted and displayed, which facilitates system managers to make decisions quickly.

[0011] Furthermore, the business database is constructed in the following manner:

[0012] A1, obtaining a number of data items that can be collected according to the data category, and sorting the data items by importance to obtain a data item list; wherein the data category includes error data, performance data, and behavior data;

[0013] A2, divide the life cycle of e-commerce business into several business stages, and set the data collection ratio of each data category according to the business stage to obtain several data collection ratios;

[0014] A3, using database technology, storing several business stages and corresponding several data collection ratios in a database to obtain a business database.

[0015] By dividing the e-commerce business life cycle into several business stages and setting different data collection ratios according to the characteristics of each stage, we can ensure that closely related data is collected at each specific stage, which helps to improve the efficiency and accuracy of data analysis.

[0016] Furthermore, the data items that can be collected according to the data category include:

[0017] The data items of error data include: error type, error time, error location, error frequency, and error recovery time;

[0018] The data items of performance data include: home page loading time, product details page loading time, shopping cart page loading time, payment page loading time, page rendering time, server response peak time, API call time, CPU usage, memory usage, network bandwidth usage;

[0019] The data items of behavioral data include: link pasting source, browsing time, click frequency, search history, number of shopping cart products, order time, payment time, comment rate, sharing rate, number of logins, online time, product collection frequency, product dwell time, and repeat purchase cycle.

[0020] Collecting rich big data covering three categories: error data, performance data, and behavioral data, and comprehensively collecting various types of data can fully present the operating status of e-commerce business from multiple dimensions, providing sufficient data foundation for in-depth analysis of user experience, system stability, and business conversion, and helping to discover the correlations and potential problems hidden between different data.

[0021] Furthermore, starting data embedding to collect data according to the matching results includes:

[0022] B1, receiving the current business stage of the system from the server, and obtaining the current business stage;

[0023] B2, connect to the business database, match the corresponding fields in the business database according to the current business stage, and obtain several data collection ratios of the current business stage;

[0024] B3, construct a tracking point table according to the data item list, and obtain a tracking point table including tracking point ID, event name, trigger condition, collected data field and data category;

[0025] B4, start the corresponding data collection operation in the tracking table according to the data collection ratio to obtain some initial data.

[0026] Constructing a tracking point table based on a list of data items makes the data collection process more standardized and regular. This not only makes it easier for staff to clearly understand the specific function and collection content of each tracking point, but also facilitates the subsequent management, analysis, and troubleshooting of the collected data.

[0027] Furthermore, the multi-threaded preprocessing of the initial data using the edge computing nodes includes:

[0028] C1, set up a number of edge servers and a number of IDs according to the number of data categories, and deploy a number of edge computing nodes according to the number of edge server cores;

[0029] C2, dynamically associates a number of IDs with a number of data categories according to a number of edge computing nodes in a number of edge servers and matching results;

[0030] C3, according to a number of IDs, transmits a number of initial data to the corresponding edge server, and configures a message queue service for the corresponding edge computing node to obtain a number of message queues;

[0031] C4, starts a multi-thread mechanism in the message queue and performs multi-thread preprocessing on some initial data; the preprocessing includes data cleaning, data conversion, and data encoding.

[0032] Several edge servers are set up according to the number of data categories to ensure that each edge server can reasonably undertake a part of the data processing tasks, give full play to the computing power of each edge server, avoid idle or overused resources, and improve the utilization rate of computing resources of the entire system. At the same time, the multi-threaded processing method allows multiple data processing tasks to be carried out simultaneously, greatly shortening the time of data preprocessing and improving the overall efficiency of data preprocessing.

[0033] Furthermore, dynamically associating the plurality of IDs with the plurality of data categories according to the plurality of edge computing nodes in the plurality of edge servers and the matching results includes:

[0034] C3-1, multiply the data collection ratio in the matching result by the number of data items in the corresponding data category and round down to obtain the number of data items in several data categories;

[0035] C3-2, sorting a number of edge servers according to the number of edge computing nodes to obtain an edge server sorting table;

[0036] C3-3, sorting several data categories according to the number of data items to obtain a data category sorting table;

[0037] C3-4, the edge server sorting table and the data category sorting table are matched one by one to obtain a number of associated IDs and corresponding data categories.

[0038] By dynamically associating edge servers and data categories, it is ensured that data can be accurately matched with corresponding edge servers and computing nodes during transmission and processing; at the same time, the data is reasonably diverted to the computing nodes on each edge server for processing, so that the load of each computing node is relatively balanced, further improving the efficiency of the entire data processing process and avoiding the problem of some nodes being overloaded and affecting the processing speed.

[0039] Furthermore, the pre-trained general model is constructed based on deep learning, including:

[0040] D1, collect and preprocess the historical data of several data items, perform data annotation and feature engineering on the preprocessed historical data and integrate them into the data warehouse to obtain an integrated data set;

[0041] D2, building a multimodal data processing model based on the Transformer architecture;

[0042] D3, divide the integrated data set into training set, validation set and test set according to the preset ratio;

[0043] D4, use the training set and validation set to train and validate the multimodal data processing model, and iteratively optimize the multimodal data processing model according to the loss function and optimization algorithm to obtain the multimodal data processing model with the best validation accuracy;

[0044] D5, use the test set to perform performance testing on the multimodal data processing model with the best verification accuracy, obtain the test accuracy, and determine whether the test accuracy reaches the preset threshold; if yes, save the multimodal data processing model with the best verification accuracy to obtain the pre-trained general model; if no, adjust the parameters of the multimodal data processing model and repeat D4-D5; the input of the general model is several data items, and the output is user introduction intention category, user retention category, user conversion category and user loyalty category.

[0045] The universality of the model allows it to be applied to multiple stages and different scenarios of e-commerce business, without the need to rebuild a new model for each specific stage or scenario. When fine-tuning specific stages in the future, it can also be quickly adjusted and adapted based on this universal model, which greatly saves the time and cost of model development, while also ensuring the coherence and consistency of the entire data analysis system.

[0046] Furthermore, the analysis model of the corresponding stage is obtained after fine-tuning based on the pre-trained general model, including:

[0047] E1, collect some initial data after preprocessing in the corresponding stage, and perform data annotation, feature engineering and integration to obtain a specific data set;

[0048] E2, fine-tuning the general model according to the data dimensions of the initial data preprocessed in the corresponding stage to obtain a fine-tuned specific model;

[0049] E3, input the specific data set into the specific model for training, verification and testing, and obtain the analysis model of the corresponding stage whose test accuracy meets the preset test threshold.

[0050] Fine-tuning the general model according to the data dimensions of several initial data preprocessed in the corresponding stage can flexibly cope with the differences in data dimensions at different stages. At different stages of e-commerce business, the data dimensions of concern may be different. For example, the user introduction stage may focus more on dimensions such as user source channels, while the user conversion stage focuses on dimensions such as shopping cart operations and payment information. By fine-tuning the model structure and parameters to adapt to changes in these data dimensions, it can ensure that the model can effectively process input data at each stage, and improve the model's adaptability and analysis effect to data at different stages.

[0051] Compared with the prior art, the present invention has the following beneficial effects:

[0052] By dividing the e-commerce business life cycle into multiple business stages, setting different data collection ratios according to the characteristics of each stage, and building a data collection table based on the data item list, the accuracy and standardization of data collection are ensured, blind collection is avoided, and the collected data can better reflect the business characteristics and needs of each stage, which is also convenient for subsequent management, analysis and problem troubleshooting;

[0053] In terms of data processing, edge servers are set up according to the number of data categories to give full play to the computing power of each edge server and improve the utilization rate of computing resources. Edge computing nodes are used for multi-threaded preprocessing, and the multi-threaded mechanism is started in the message queue, which greatly shortens the data preprocessing time and improves the overall efficiency. In addition, the edge server and data category are dynamically associated to ensure accurate data matching and reasonable diversion, so that the computing node load is balanced, further optimizing the data processing process.

[0054] The pre-trained general model is based on deep learning and has strong versatility. It can be applied to multiple stages and scenarios of e-commerce business, saving model development time and cost, and ensuring the coherence and consistency of the data analysis system. The analysis model of the corresponding stage is fine-tuned based on the general model, which improves the model's adaptability and analysis effect to data at each stage, and can deeply explore the relationship between data, thereby achieving accurate data analysis.

[0055] The result display module classifies and counts the data and analysis results and displays them in the monitoring center, making it easy for staff to quickly obtain information. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0057] Figure 1 A schematic diagram of a process flow of a data analysis and display system based on a large model provided by the present invention;

[0058] Figure 2 A schematic diagram of a framework of a data analysis and display system based on a large model provided by the present invention;

[0059] Figure 3 A schematic diagram of the flow of a data processing module in a data analysis and display system based on a large model provided by the present invention;

[0060] Figure 4 A schematic diagram of the construction process of an analysis model at a corresponding stage in a large-model-based data analysis and display system provided by the present invention. DETAILED DESCRIPTION

[0061] The technical solution of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0062] See also Figure 1-Figure 4 The first embodiment of the present invention provides a data analysis and display system based on a large model, including:

[0063] Data collection module: used to match the current business stage with the business database to obtain matching results, and start data collection based on the matching results to obtain some initial data; the matching results indicate the current business stage and the data collection ratio, and the data collection ratio indicates the ratio of the number of collected data items to the data items included in each data category;

[0064] Data processing module: used to deploy several edge computing nodes on several edge servers, and use several edge computing nodes to perform multi-threaded preprocessing on several initial data to obtain several preprocessed initial data;

[0065] Data analysis module: used to transmit some pre-processed initial data to the monitoring center, call the analysis model of the corresponding stage according to the current business stage, and pass some pre-processed initial data to the analysis model of the corresponding stage to obtain the analysis result; wherein, the analysis model of the corresponding stage is obtained after fine-tuning based on the pre-trained general model, and the pre-trained general model is constructed based on deep learning;

[0066] Result display module: used to classify and count the initial data and analysis results after preprocessing, obtain several data reports and display them in the monitoring center;

[0067] It should be noted that the data acquisition module, data processing module, data analysis module and result display module in the system are in communication connection, and the input of the result display module includes the data processing module and the data analysis module.

[0068] In the e-commerce business system, with the increasingly fierce market competition and the continuous changes in consumer demand, enterprises need to have a deeper and more accurate understanding and grasp of their own business in order to achieve sustainable development and optimization. The data analysis and display system based on the big model provided by the present invention is of vital significance in meeting this core demand.

[0069] In this embodiment, the data collection module is used to start data point collection by matching the current business stage with the business database, which can ensure that the appropriate number and type of data items closely related to the business at each stage are obtained, avoid collecting irrelevant or redundant data, and provide a solid foundation for subsequent accurate analysis.

[0070] Specifically, we first need to build a business database, including:

[0071] A1, determine and sort data items:

[0072] First, determine the data items that can be collected based on the data category (error data, performance data, behavior data). For error data, its data items include but are not limited to error type, error time, error location, error frequency, error recovery time, etc.; performance data include but are not limited to home page loading time, product details page loading time, shopping cart page loading time, payment page loading time, page rendering time, server response peak time, API call time, CPU usage, memory usage, network bandwidth usage, etc.; behavior data include but are not limited to, link paste source, browsing time, click frequency, search history, shopping cart product quantity, order time, payment time, comment rate, sharing rate, number of logins, online time, product collection frequency, product stay time, repeated purchase cycle, etc.;

[0073] Then, the expert evaluation method is used to sort these data items by importance to form a list of data items. For example, the list of data items of sorted behavioral data is: number of shopping cart products, order time, payment time, repeated purchase cycle, browsing time, product dwell time, click frequency, search history, product collection frequency, number of logins, online time, comment rate, sharing rate, link paste source;

[0074] A2, Business stage division and proportion setting:

[0075] Divide the life cycle of e-commerce business into multiple business stages, such as user introduction stage, user retention stage, user conversion stage, and user loyalty cultivation stage; then set the data collection ratio of each data category according to the characteristics and needs of each business stage; for example, in the user conversion stage, in the behavioral data category, more attention is paid to the user's purchase behavior, then the collection ratio can be set to 60%, that is, the first 60% of the data item list is collected, and the data items obtained include: shopping cart product quantity, order time, payment time, repeated purchase cycle, browsing time, product dwell time, click frequency, search history, product collection frequency;

[0076] A3, database storage:

[0077] Using database technology, several divided business stages and the corresponding data collection ratios are stored in the database to build a business database.

[0078] Then, the current business stage is matched with the business database to obtain the matching result, and data collection is started based on the matching result; including:

[0079] B1, get the current business stage:

[0080] The data collection module receives information about the current business stage of the system from the server. For example, when an e-commerce platform is promoting new user registration, the server determines that the current business stage is the user introduction stage and sends this information to the data collection module;

[0081] B2, matching data collection ratio:

[0082] Connect to the previously constructed business database, search and match the corresponding fields in the database according to the received current business stage, and thus obtain the data collection ratio of each data category in the current business stage;

[0083] B3, build a tracking table:

[0084] Construct a tracking point table based on the data item list obtained above; the tracking point table contains a tracking point ID (used to uniquely identify each tracking point), an event name (such as "user order event"), a trigger condition (such as "triggered when the user clicks the confirmation payment button"), a collected data field (such as order time, payment time, etc.), and a data category (such as behavioral data); wherein the collected data field is related to the data item in the data item list;

[0085] B4, start data collection:

[0086] Start the corresponding data collection operation in the tracking table according to the acquired data collection ratio. For example, in the user retention stage, start the collection of data items such as browsing time and number of logins in the behavior data and some data items such as home page loading time in the performance data according to the set collection ratio. The collected data are some initial data, which will be used for subsequent data processing and analysis.

[0087] In today's fast-paced business environment, the value of data is often closely related to its timeliness. Efficient data processing can ensure that enterprises can obtain and analyze big data in a timely manner, so as to respond quickly. At the same time, the scale of big data is huge, and the processing process requires a large amount of computing resources, storage resources, and network bandwidth. Efficient data processing technology can improve the utilization rate of the entire system resources and reduce operating costs.

[0088] In this embodiment, the data processing module is used to call the edge computing node on the edge server to perform multi-threaded preprocessing;

[0089] Specifically, first deploy edge servers and edge computing nodes:

[0090] First, the number of data categories is counted, and the number of edge servers is determined based on this number, and a unique ID is assigned to each server. In this embodiment, the number of data categories is 3, namely error data, performance data, and behavior data. It is assumed that the obtained IDs are: the ID of edge server 1 is 1, the ID of edge server 2 is 2, and the ID of edge server 3 is 3.

[0091] Next, check the number of cores of each edge server; for example, edge server 1 has 2 cores, edge server 2 has 6 cores, and edge server 3 has 4 cores; deploy edge computing nodes based on the number of cores, such as deploying 4 edge computing nodes on edge server 1, 12 edge computing nodes on edge server 2, and 8 edge computing nodes on edge server 3;

[0092] Then dynamically associate the data category with the edge server, so that one edge server can process one category of data:

[0093] C3-1, the data collection ratio is multiplied by the number of data items of the corresponding data category and rounded down to obtain the number of data items of several data categories; for example, it can be seen from the above that in this embodiment, the error data includes 5 data items, the performance data includes 10 data items, and the behavior data includes 14 data items; assuming that in the user conversion stage, the proportion of error data is 50%, the proportion of performance data is 50%, and the proportion of behavior data is 60%, then the number of data items in each category is 3, 5, and 9 respectively;

[0094] C3-2, sorting several edge servers according to the number of edge computing nodes, and obtaining an edge server sorting table: edge server 1, edge server 3, edge server 2;

[0095] C3-3, sorting several data categories according to the number of data items, and obtaining a data category sorting table as follows: error data, performance data, and behavior data;

[0096] C3-4, the edge server sorting table is matched with the data category sorting table one by one, and the association result is: edge server 1 processes error data, edge server 3 processes performance data, and edge server 2 processes behavior data;

[0097] Then, according to the association results, the ID is used to transfer some initial data to the corresponding edge server; and the message queue service is configured for the corresponding edge computing node on each edge server. These message queues are used to cache and manage the data to be processed to ensure the orderliness and efficiency of data processing;

[0098] Finally, the multi-thread mechanism is started in the message queue. For example, in the message queue for processing behavior data on edge server 2, multiple threads are started to perform multi-thread preprocessing on the data, including data cleaning (such as removing duplicate payment times, invalid links, etc.), data conversion (such as converting the time format to a specific format), and data encoding (such as encoding the product category of the search record). Each thread processes a part of the data independently, which greatly improves the speed of data processing.

[0099] Next, in the data analysis module, the pre-processed data is transmitted to the monitoring center, and the analysis model of the corresponding stage is called according to the current business stage. The pre-processed initial data is passed into the analysis model of the corresponding stage to obtain the analysis result; wherein, the analysis model of the corresponding stage is obtained after fine-tuning based on the pre-trained general model;

[0100] Pre-trained general models are built on deep learning, including:

[0101] D1, collect and preprocess the historical data of several data items, perform data annotation and feature engineering on the preprocessed historical data and integrate them into the data warehouse to obtain an integrated data set;

[0102] D2, building a multimodal data processing model based on the Transformer architecture;

[0103] D3, divide the integrated data set into training set, validation set and test set according to the preset ratio;

[0104] D4, use the training set and validation set to train and validate the multimodal data processing model, with the goal of predicting the user's introduction results, retention results, conversion results, and loyalty level, and iteratively optimize the multimodal data processing model based on the loss function and optimization algorithm to obtain the multimodal data processing model with the best verification accuracy;

[0105] D5, using the test set to perform a performance test on the multimodal data processing model with the best verification accuracy, obtain the test accuracy, and determine whether the test accuracy reaches a preset threshold; if yes, save the multimodal data processing model with the best verification accuracy to obtain a pre-trained general model; if no, adjust the parameters of the multimodal data processing model and repeat D4-D5;

[0106] The input of the general model is a number of data items, and the output is user introduction intention category, user retention category, user conversion category, and user loyalty category;

[0107] Moreover, the categories of user introduction include: “high-intent introduction” (e.g., user registration, browsing multiple product pages), “medium-intent introduction” (only browsing some pages, not registered but staying for a long time), and “low-intent introduction” (quickly leaving the platform);

[0108] User retention categories include: "retained users" (continuously active over a period of time, such as logging in, browsing, purchasing, etc.) and "churned users" (not logged in for a period of time or very low activity);

[0109] User conversion categories include: "Converted users" (successfully placed an order and paid) and "Unconverted users" (added to the shopping cart but did not pay, not added to the shopping cart, etc.);

[0110] User loyalty categories include: "high loyalty users", "medium loyalty users" and "low loyalty users" (based on repeated purchase cycles).

[0111] After obtaining the pre-trained general model, fine-tune the general model according to the dimension size of the data collected in the corresponding stage;

[0112] Specifically, the data acquisition module and the data processing module are used to collect a number of initial data after preprocessing in the corresponding stage, and then data annotation, feature engineering and integration are performed to obtain a specific data set;

[0113] Then determine the data dimension after preprocessing in the corresponding stage and compare it with the input dimension of the pre-trained general model. Adjust the input layer of the general model to reduce the number of input nodes to match the data in the current stage. At the same time, adjust the parameters of the hidden layer appropriately, such as reducing the number of neurons in certain layers or simplifying the connection structure to avoid overfitting caused by excessive model complexity.

[0114] Similarly, a specific data set is divided into a training set, a validation set, and a test set in a certain ratio; and the specific data set is input into a specific model for training, validation, and testing to obtain an analysis model of the corresponding stage whose test accuracy meets a preset test threshold; that is, for each stage, the above steps are repeated to obtain a user introduction analysis model, a user retention analysis model, a user conversion analysis model, and a user loyalty cultivation model, and the output result is also an output subset of the pre-trained general model, thereby obtaining the corresponding analysis result.

[0115] Finally, in the result display module, the pre-processed initial data and analysis results are classified and counted; for example, in the error data category, the number of occurrences of different error types, the distribution of error occurrence time, etc. are counted; for performance data, the average, maximum, and minimum values ​​of various page loading times, the distribution of server response peak time, etc. are calculated; in terms of behavioral data, the browsing time distribution of different user groups (such as classified according to user loyalty), the average number of shopping cart products, the order time distribution, etc. are counted. For the analysis results, classified statistics are performed based on the output of different analysis models (such as user introduction analysis model, user retention analysis model, etc.). For example, the number and proportion of users with different intentions to introduce (high, medium, and low intentions), the changing trend of the ratio of retained users to lost users, the conversion rate of converted users and non-converted users in different time periods, etc. are counted;

[0116] Then, several data reports are generated based on the results of the classification statistics and displayed on the interface of the monitoring center; for example, an error data report is generated, which contains fields such as error type, error frequency, and error time trend; a performance data report covers page loading time statistics, server response performance indicators, etc.; a behavior data analysis report displays the behavior patterns and key indicators of different user groups; a user conversion analysis report presents the conversion rate and conversion user characteristics at different stages (such as daily and weekly); a user retention analysis report displays the changes in retained users and lost users and analysis of related factors, etc.

[0117] Working principle of the present invention:

[0118] First, the data collection module matches the current business stage with the business database to determine the data collection ratio, and then starts the corresponding data collection points to obtain some initial data;

[0119] Next, the data processing module deploys edge computing nodes on the edge server and uses multi-threading technology to pre-process the initial data to improve the efficiency of data processing. During the pre-processing process, the data categories and edge computing nodes are dynamically associated according to the number of data categories and the computing power of the edge server to ensure reasonable data diversion and load balancing.

[0120] Then, the data analysis module transmits the preprocessed data to the monitoring center and calls the corresponding analysis model for analysis according to the current business stage. These analysis models are obtained after fine-tuning the pre-trained general model. The general model itself is built based on deep learning. It collects historical data, performs annotation, feature engineering and integration, builds a multimodal data processing model, and is optimized through training and verification. The fine-tuning process adjusts the general model according to the data dimensions of a specific stage to meet the data analysis needs of different business stages;

[0121] Finally, the result display module classifies and counts the preprocessed data and analysis results, generates data reports and displays them in the monitoring center to facilitate system managers to make quick decisions.

[0122] The above embodiments are only used to illustrate the technical method of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical method of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical method of the present invention.

Claims

1. A data analysis and display system based on a large model, characterized in that: include: Data collection module: used to match the current business stage with the business database to obtain matching results, and start data collection based on the matching results to obtain some initial data; the matching results indicate the current business stage and the data collection ratio, and the data collection ratio indicates the ratio of the number of collected data items to the data items included in each data category; Data processing module: used to deploy several edge computing nodes on several edge servers, and use several edge computing nodes to perform multi-threaded preprocessing on several initial data to obtain several preprocessed initial data; Data analysis module: used to transmit the pre-processed initial data to the monitoring center, call the analysis model of the corresponding stage according to the current business stage, and pass the pre-processed initial data to the analysis model of the corresponding stage to obtain the analysis result; wherein, the analysis model of the corresponding stage is obtained after fine-tuning based on the pre-trained general model, and the pre-trained general model is constructed based on deep learning; Result display module: used to classify and count the initial data and analysis results after preprocessing, obtain several data reports and display them in the monitoring center.

2. A data analysis and display system based on a large model according to claim 1, characterized in that: The business database is constructed in the following manner: A1, obtaining a number of data items that can be collected according to the data categories, and sorting the data items by importance to obtain a data item list; wherein the data categories include error data, performance data, and behavior data; A2, divide the life cycle of e-commerce business into several business stages, and set the data collection ratio of each data category according to the business stage to obtain several data collection ratios; A3, using database technology, storing several business stages and corresponding several data collection ratios in a database to obtain a business database.

3. A data analysis and display system based on a large model according to claim 2, characterized in that: The data items that can be collected according to the data category include: The data items of error data include: error type, error time, error location, error frequency, and error recovery time; The data items of performance data include: home page loading time, product details page loading time, shopping cart page loading time, payment page loading time, page rendering time, server response peak time, API call time, CPU usage, memory usage, network bandwidth usage; The data items of behavioral data include: link pasting source, browsing time, click frequency, search history, number of shopping cart products, order time, payment time, comment rate, sharing rate, number of logins, online time, product collection frequency, product dwell time, and repeat purchase cycle.

4. A data analysis and display system based on a large model according to claim 1, characterized in that: The starting of data embedding for data collection according to the matching results includes: B1, receiving the current business stage of the system from the server, and obtaining the current business stage; B2, connect to the business database, match the corresponding fields in the business database according to the current business stage, and obtain several data collection ratios of the current business stage; B3, construct a tracking point table according to the data item list, and obtain a tracking point table including tracking point ID, event name, trigger condition, collected data field and data category; B4, start the corresponding data collection operation in the tracking table according to the data collection ratio to obtain some initial data.

5. The data analysis and display system based on a large model according to claim 1 is characterized in that: The method of using a plurality of edge computing nodes to perform multi-threaded preprocessing on a plurality of initial data includes: C1, set up a number of edge servers and a number of IDs according to the number of data categories, and deploy a number of edge computing nodes according to the number of edge server cores; C2, dynamically associates a number of IDs with a number of data categories according to a number of edge computing nodes in a number of edge servers and matching results; C3, according to a number of IDs, transmits a number of initial data to the corresponding edge server, and configures a message queue service for the corresponding edge computing node to obtain a number of message queues; C4, starts the multi-thread mechanism in the message queue and performs multi-thread preprocessing on some initial data.

6. A data analysis and display system based on a large model according to claim 5, characterized in that: The dynamically associating the plurality of IDs with the plurality of data categories according to the plurality of edge computing nodes in the plurality of edge servers and the matching results includes: C3-1, multiply the data collection ratio in the matching result by the number of data items in the corresponding data category and round down to obtain the number of data items in several data categories; C3-2, sorting a number of edge servers according to the number of edge computing nodes to obtain an edge server sorting table; C3-3, sorting several data categories according to the number of data items to obtain a data category sorting table; C3-4, the edge server sorting table and the data category sorting table are matched one by one to obtain a number of associated IDs and corresponding data categories.

7. The data analysis and display system based on a large model according to claim 1, characterized in that: The pre-trained general model is built based on deep learning and includes: D1, collect and preprocess the historical data of several data items, perform data annotation and feature engineering on the preprocessed historical data and integrate them into the data warehouse to obtain an integrated data set; D2, building a multimodal data processing model based on the Transformer architecture; D3, divide the integrated data set into training set, validation set and test set according to the preset ratio; D4, use the training set and validation set to train and validate the multimodal data processing model, and iteratively optimize the multimodal data processing model according to the loss function and optimization algorithm to obtain the multimodal data processing model with the best validation accuracy; D5, use the test set to perform performance testing on the multimodal data processing model with the best verification accuracy, obtain the test accuracy, and determine whether the test accuracy reaches the preset threshold; if yes, save the multimodal data processing model with the best verification accuracy to obtain the pre-trained general model; if no, adjust the parameters of the multimodal data processing model and repeat D4-D5.

8. The data analysis and display system based on a large model according to claim 1, characterized in that: The analysis model of the corresponding stage is obtained after fine-tuning based on the pre-trained general model, including: E1, collect some initial data after preprocessing in the corresponding stage, and perform data annotation, feature engineering and integration to obtain a specific data set; E2, fine-tuning the general model according to the data dimensions of the initial data preprocessed in the corresponding stage to obtain a fine-tuned specific model; E3, input the specific data set into the specific model for training, verification and testing, and obtain the analysis model of the corresponding stage whose test accuracy meets the preset test threshold.

Images