Cloud edge-end collaborative user behavior analysis system based on large language model

Abstract

A user behavior analysis system based on cloud side-end cooperation of a large language model is used for carrying out information collection, cloud side-end cooperation and user behavior analysis on mobile phones and computer equipment, and comprises an information collection module, a cloud side-end cooperation module and a user behavior analysis module. According to the invention, information acquisition is carried out on electronic data of mobile phone equipment, information privacy collaboration is carried out through the cloud side end module, and then user behavior analysis is carried out based on a large model, so that the defects of low collaboration efficiency, high user behavior identification false alarm rate and the like of an existing user behavior analysis system are overcome; the evidence obtaining efficiency and the user behavior analysis effect are effectively improved by introducing a cloud side-end cooperation and large language model, and the cloud side-end cooperation behavior analysis system which is high in efficiency, high in precision and light in weight is achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of Internet data mining, and specifically to a user behavior analysis system based on cloud-edge-end collaboration based on a large language model. Background Art

[0002] In recent years, with the vigorous development of big language model technology and the profound changes in the fields of big data and artificial intelligence Internet of Things, the cloud-edge-end collaborative user behavior analysis system based on big language model is leading the wave of a new generation of data analysis and insights. This system is not only large-scale and comprehensive, but its application territory is expanding at an unprecedented rate, becoming an intelligent bridge connecting users, services and decisions. With the support of big language model technology, the system can deeply explore the potential patterns of user behavior and provide unimaginable personalized experience and service optimization. However, when facing the torrent of Internet-scale data, the ever-changing user behavior and the demanding requirements for instant response, traditional single systems are limited by the centralized deployment of computing resources, the delay of data processing and the limitations of intelligence. It is in this context that the cloud-edge-end collaborative architecture based on big language model was designed and implemented, which represents a new height of technological development and aims to overcome the above challenges.

[0003] In view of this, we propose a user behavior analysis system based on cloud-edge-end collaboration with a large language model. Summary of the invention

[0004] In order to solve the above technical problems, the present invention proposes a user behavior analysis system based on cloud-edge-end collaboration with a large language model, which solves the technical problems raised in the above background technology, and realizes the generalization ability, context understanding and generation ability of the large language model. Through the close cooperation of cloud, edge and terminal, it realizes the real-time collection, efficient preprocessing and in-depth analysis of user behavior data; through the application of large language model technology, the system can more accurately understand user intentions and predict behavior trends.

[0005] To achieve the above object, the technical solution adopted by the present invention is:

[0006] A user behavior analysis system based on cloud-edge-end collaboration with a large language model, including a device information collection module, a cloud-edge-end collaboration module and a user behavior analysis module:

[0007] Device information collection module: By establishing a secure connection with the mobile device, it obtains hardware information and application data, and creates a local database on the edge server for data backup; it merges the collected information, encrypts it according to preset rules, generates an application database key, and implements encrypted storage and on-demand decoding access of data;

[0008] Cloud-edge-end collaboration module: Full-duplex communication is achieved through Websocket and MQTT, ensuring real-time information synchronization and business collaboration between cloud, edge and end, including device authentication, remote management and autonomy; promoting data privacy and secure synchronization to the cloud for efficient analysis, accelerating the agile analysis of user behavior using large language models;

[0009] User behavior analysis module: includes a large language model, which uses the trained model to predict new data; and analyzes information in mobile applications through a large language model.

[0010] As a further improvement of the present invention, the device information collection module collects data applied in the mobile device, specifically including:

[0011] a.1. Connect the device to be collected to the system, establish a link between the information collection module and the mobile device, obtain the hardware information of the device to be collected, collect application data, and generate a local database on the edge side to back up the application data;

[0012] a.2. The information acquisition module combines the hardware information and the application information, encrypts them according to the rules of the application database, obtains the key of the application database, and then decodes the database to obtain the application data.

[0013] As a further improvement of the present invention, the cloud-edge-end collaboration module is used to provide information and service collaboration between the collected mobile device and the edge node, and between the edge node and the cloud platform, specifically including:

[0014] b.1. Full-duplex communication based on Websocket and MQTT completes cloud-edge communication, supports synchronous communication between cloud and edge nodes based on Web applications, and obtains hardware information and application information of mobile devices in real time; supports the cloud-edge-end integrated system framework to authenticate and authorize edge devices, and implements effective management of edge devices; supports information collaboration between edge and cloud;

[0015] b.2. The edge side of the cloud-edge-device collaboration module provides edge computing business modules, which include edge container management, edge node management, mobile device access management, large language model management, and edge OS.

[0016] The platform side of the cloud-edge-device collaboration module provides a cloud-edge-device collaboration console, edge device instance management, and edge device driver management;

[0017] The cloud-edge-end collaboration module provides capability support for upstream applications, including user portrait generation applications and user portrait analysis applications; after the device information is collected, the cloud-edge-end collaboration module synchronizes the information privacy to the cloud for efficient analysis.

[0018] As a further improvement of the present invention, the user behavior analysis module builds a large language model based on the LLaMa model that can analyze the user's mobile phone application information, specifically including:

[0019] c.1. At the network level of the large language model, the LLaMa model has been improved:

[0020] c.2. The pre-training data level includes the following:

[0021] c.2.1. Integrate multi-dimensional dialogue data annotation technology to cover multiple aspects of two-person and multi-person dialogue understanding, build comprehensive and rich training resources, and enhance the system's understanding of complex dialogue structures;

[0022] c.2.2. Use fine-grained label annotation to help the model learn language expressions and interaction strategies that are closer to human communication habits, and optimize the model's performance on various auxiliary tasks;

[0023] c.2.3. Use an exclusion mechanism to ensure that the text analysis system has a high degree of generalization ability and can adapt to various non-standard communication scenarios;

[0024] c.2.4. Strengthen the dangerous behavior mining mechanism, pay attention to and mark potential violence, prejudice and sensitive content, and improve the system's sensitivity to dangerous content;

[0025] c.2.5. Use ChatML format as the standard for data organization and exchange to accurately analyze the dialogue structure, deeply explore the contextual meaning, and improve the accuracy and naturalness of the response;

[0026] c.3. Model fine-tuning based on prompt words: Structural reorganization of input text information according to preset templates to ensure that the information is processed in an optimized form, so as to maximize the potential of the pre-trained language model and effectively extract and utilize deep knowledge in the context;

[0027] c.4. Fine-tuning of model parameters based on LoRA: Ensure local and targeted adjustments to the model while maintaining the overall architecture of the model, thereby improving the performance of the model on specific tasks while retaining the generalization ability of the model.

[0028] As a further improvement of the present invention, at the network level of the large language model described in c.1, the LLaMa model is improved including the following contents:

[0029] c.1.1. Use a pre-normalization strategy to apply normalization processing before each layer of the neural network to improve the stability of the training process and the convergence efficiency of the model;

[0030] c.1.2. Use RMSNorm as the normalization method. RMSNorm calculates the root mean square value of each dimension of the input tensor and scales it accordingly, thus providing enhanced stability when processing input data with different scales.

[0031] c.1.3. Select SwiGLU as the activation function. The SwiGLU activation function combines the characteristics of the Swish function and the gated linear unit;

[0032] c.1.4. Adjust the dimensionality of the feedforward network, reducing it from four times the traditional hidden layer dimension to three-eighths of the hidden dimension, optimizing the model's storage requirements and computing resource utilization efficiency while maintaining or improving the model's learning ability and expressiveness.

[0033] As a further improvement of the present invention, the LoRA-based model parameter fine-tuning described in c.4 specifically includes:

[0034] c.4.1. The pre-trained model parameters are represented as θ. During fine-tuning, LoRA does not modify θ directly, but introduces a set of low-rank matrices A and S for each layer weight W, where A is a low-rank matrix whose size is much smaller than W and is used to capture task-specific feature changes; S is a scaling matrix used to control the intensity of the influence of A on W and maintain an appropriate scale; the fine-tuned weight W' is calculated by the following formula: W′=W+α·A·S; where α is an adjustable hyperparameter used to globally control the intensity of the influence of A on W; by adjusting the value of α, the weight between the original pre-trained weight and the task-specific adaptation is balanced;

[0035] c.4.2. Low-rank approximation: The low-rank nature of A and S means that their dimensions are much smaller than W, which greatly reduces the number of parameters that need to be trained, thereby reducing computational complexity and memory usage, making fine-tuning more efficient;

[0036] c.4.3. The number of A and S matrix elements to be optimized is far less than the number of original model parameters, which significantly reduces the computational and memory requirements during fine-tuning.

[0037] As a further improvement of the present invention, the user behavior analysis module adopts the attention mechanism calculation method to accelerate the reasoning stage, improve the calculation efficiency and memory utilization; reduce GPU storage access through data segmentation, local normalization and hardware optimization strategy, specifically including the following contents:

[0038] d.1. Data partitioning: Divide the query matrix Q, key matrix K and value matrix V into T along the sequence length dimension c non-overlapping blocks, each block size is B*d, B = [L / T c ]; where L is the sequence length and d is the feature dimension;

[0039] d.2. Intra-block similarity calculation: For each block i, calculate the intra-block and cross-block similarity score matrix and Use the following formula:

[0040]

[0041] Among them, q im represents the mth query vector of the ith block in the query matrix, k in represents the nth key vector of the i-th block in the key matrix, k jn represents the nth key vector of the jth block in the key matrix, is the intra-block similarity score matrix; is the cross-block similarity score matrix; T represents the transpose operation, which converts the rows of the matrix into columns or converts the columns into rows so that the dot product operation can be performed;

[0042] d.3. Normalization and weighted summation within each block: Similarity matrix within each block Apply row normalization to get the attention weight matrix P ii , P ii [m,:] represents the normalized attention weight within the block, that is, the attention distribution of the mth row in the i-th block, P ii [m,:] is calculated by the following formula:

[0043]

[0044] Then, use P ii For the corresponding value block V ii Perform weighted summation to obtain the output matrix O of the block ii ;

[0045] d.4. Global normalization and output combination: If there is cross-block similarity, merge the calculation results of all blocks and perform global normalization, and finally combine the output matrix O of all blocks ii To recover the complete output matrix O, ensuring the consistency of global attention.

[0046] As a further improvement of the present invention, an attention mechanism calculation method is used to accelerate reasoning, and a dynamic memory management strategy is used to accelerate the reasoning process of a large language model; the input data is divided into multiple subsequences, each subsequence is targeted to allocate computing resources, and the PagedAttention algorithm is used to dynamically load and unload the key and value vectors of the attention mechanism to the GPU memory; by presetting the memory page size to match the GPU cache capacity, the key-value data of the current subsequence is called in as needed while clearing the memory that is no longer necessary, thereby completing the attention calculation of the large-scale language model while ensuring efficient memory utilization, and finally summarizing the reasoning results of each subsequence to produce an overall analysis output.

[0047] As a further improvement of the present invention, a dynamic memory management strategy is used to accelerate the reasoning process of a large language model, which effectively alleviates the problem of tight memory resources in an edge computing environment. Specifically, the following contents are included:

[0048] e.1. Collect the input data sequence to be inferred;

[0049] e.2. dividing the input data sequence into a plurality of continuous subsequences, each subsequence corresponding to one or more steps of model processing;

[0050] e.3. Dynamically allocate computing resources for the processing of each subsequence, and use the PagedAttention algorithm for memory management for the key-value pairs in the attention mechanism.

[0051] As a further improvement of the present invention, the memory management using the PagedAttention algorithm described in e.3 includes the following contents:

[0052] e.3.1. Predetermine a memory page size that is suitable for loading into the GPU cache at one time;

[0053] e.3.2. The keys and values ​​required for the current processing subsequence are loaded from external storage into the current memory page of GPU memory as needed, while the memory pages occupied by the keys and values ​​of the previous subsequence that are no longer needed are released to maintain memory usage efficiency;

[0054] e.3.3. Perform attention calculations using the keys and values ​​loaded into GPU memory to assign attention weights to the input data sequence.

[0055] e.3.4. Summarize the processing results of all subsequences to generate the final inference output.

[0056] One or more technical solutions provided in the technical solution of this application have at least the following technical effects or advantages:

[0057] 1. The present invention can make full use of the generalization ability, context understanding and generation ability of the large language model, and realize the real-time collection, efficient preprocessing and in-depth analysis of user behavior data through the close cooperation of cloud, edge and terminal.

[0058] 2. The cloud side serves as a powerful computing center, responsible for complex model training and big data processing; edge computing undertakes real-time analysis and decision-making tasks close to users, reducing the time cost of data going back and forth to the cloud; terminal devices focus on the initial screening of data and situational awareness. The three work together to build a behavioral analysis network that is both sensitive and powerful.

[0059] 3. The application of large language model technology, especially its integration in natural language processing, computer vision and other fields, enables the system to more accurately understand user intentions, predict behavioral trends, and provide personalized service recommendations while protecting user privacy.

[0060] 4. With the help of the self-learning and continuous optimization mechanism of the large language model, the system can dynamically adapt to the ever-changing user behavior patterns and market environment to ensure the accuracy and timeliness of the analysis.

[0061] 5. By adopting advanced technologies such as distributed training and federated learning, combined with intelligent scheduling and optimization algorithms, we will promote the cloud-edge collaborative user behavior analysis system based on large language models to become more mature and widely used. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 This is a schematic diagram of the overall structure of the user behavior analysis system with cloud-edge collaboration based on a large language model in this application. DETAILED DESCRIPTION

[0063] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments:

[0064] Reference Figure 1 The embodiment of the present application provides a user behavior analysis system based on cloud-edge-end collaboration with a large language model. It is oriented to the user portrait construction and behavior analysis scenarios, collects information from mobile devices and transmits it privately in cloud-edge collaboration, and finally conducts behavior analysis on user information; specifically, it includes: device information collection module, cloud-edge-end collaboration module and user behavior analysis module. The cloud side is mainly composed of the user behavior analysis module; the edge node is mainly composed of the cloud-edge-end collaboration module and the device information collection module; the device side is mainly composed of the mobile devices to be collected.

[0065] Device information collection module: By establishing a secure connection with the mobile device, hardware information and application data are obtained, and a local database is created on the edge server for data backup; then, the collected information is merged and encrypted according to preset rules to generate an application database key, realizing encrypted storage and on-demand decoding access of the data, ensuring the comprehensiveness and security of data collection.

[0066] Cloud-edge-end collaboration module: Full-duplex communication is achieved with the help of Websocket and MQTT to ensure real-time information synchronization and business collaboration between cloud, edge and end, including device authentication, remote management and autonomy. This module deploys lightweight services on the edge, covering edge computing, container management to specific domain OS. At the same time, the platform side provides a unified console and rich device management functions to support upper-level applications (such as user portrait generation and analysis). It not only promotes the secure synchronization of data privacy to the cloud for efficient parsing, but also accelerates the agile analysis of user behavior using large language models, and overall improves the system's response speed and decision-making efficiency.

[0067] User behavior analysis module: including the large language model, which is divided into the training phase and the inference phase. Model inference (Inference) refers to the process of using the trained model to predict or generate new data after the training is completed. The large language model is used to analyze the information in the mobile applications of the user's authorized mobile phone. The user behavior analysis module builds a large language model based on the LLaMa model that can analyze the user's mobile application information.

[0068] Furthermore, the device information collection module is used to collect data applied in mobile devices, specifically:

[0069] a.1. Connect the device to be collected to the system, establish a link between the information collection module and the mobile device, obtain the hardware information of the device to be collected, collect application data, and generate a local database on the edge side to back up the application data;

[0070] a.2. The information acquisition module combines the hardware information and the application information, encrypts them according to the rules of the application database, obtains the key of the application database, and then decodes the database to obtain the application data.

[0071] Furthermore, the cloud-edge-end collaboration module is used to provide information and business collaboration between the collected mobile devices and edge nodes, and between edge nodes and cloud platforms, including:

[0072] b.1. Full-duplex communication based on Websocket (network socket) and MQTT (message queue telemetry transmission) completes cloud-edge communication, supports synchronous communication between cloud and edge nodes based on Web applications, and obtains mobile device hardware information and application information in real time; supports the cloud-edge integrated system framework to authenticate and authorize edge devices, and implements effective management of edge devices; supports information collaboration between edge and cloud, remote operation and maintenance, and offline autonomy;

[0073] b.2. The edge side of the cloud-edge-device collaboration module provides edge computing business modules, which include edge container management, edge node management, mobile device access management, large language model management, and edge OS.

[0074] Edge container management: responsible for managing and scheduling containerized applications running on edge nodes.

[0075] Edge node management: monitor, configure, and manage individual nodes in the edge computing network.

[0076] Mobile device access management: Manage the access and authentication of mobile devices to ensure data security and integrity.

[0077] Large language model management: Deploy and manage large language models to support tasks such as user behavior analysis and text processing.

[0078] Edge OS: Provides basic operating system support for edge computing environments to ensure device stability and performance.

[0079] The platform side of the cloud-edge-device collaboration module provides a cloud-edge-device collaboration console, edge device instance management, and edge device driver management;

[0080] The cloud-edge-end collaboration module provides capability support for upstream applications, including user portrait generation applications and user portrait analysis applications; after the device information is collected, the cloud-edge-end collaboration module can synchronize information privacy to the cloud for efficient analysis; through integrated cloud-edge-end information collaboration, user behavior can be analyzed more agilely and efficiently based on a large language model.

[0081] Furthermore, the user behavior analysis module analyzes the information in the mobile phone applications of the user's authorized mobile phone through the large language model. The user behavior analysis module builds a large language model based on the LLaMa model that can analyze the user's mobile phone application information, specifically including:

[0082] c.1. At the network level of the large language model, the LLaMa model has been improved:

[0083] i. Use a pre-normalization strategy to apply normalization processing before each layer of the neural network to improve the stability of the training process and the convergence efficiency of the model;

[0084] ii. Use RMSNorm (Root Mean Square Layer Normalization) as the normalization method instead of the traditional LayerNorm. RMSNorm calculates the root mean square value of each dimension of the input tensor and scales it accordingly, thereby providing enhanced stability when processing input data with different scales, which is particularly suitable for the training environment of large-scale deep learning models;

[0085] iii. SwiGLU is selected as the activation function, which combines the characteristics of the Swish function and the gated linear unit. Experiments have shown that it outperforms other common activation functions such as GeLU in model performance;

[0086] iv. Adjust the dimensionality of the feedforward network (FFN) from four times the traditional hidden layer dimension to three eighths of the hidden dimension, aiming to optimize the model's storage requirements and computing resource utilization efficiency while maintaining or improving the model's learning ability and expressiveness;

[0087] c.2. At the pre-training data level:

[0088] i. Integrate multi-dimensional dialogue data annotation technology, covering multiple aspects of two-person and multi-person dialogue understanding, build comprehensive and rich training resources, go beyond the standard two-person dialogue model, and enhance the system's understanding of complex dialogue structures;

[0089] ii. Through fine-grained label annotation, the model is encouraged to learn language expressions and interaction strategies that are closer to human communication habits, and the performance of the model on various auxiliary tasks is optimized;

[0090] iii. Design and implement an exclusion mechanism to avoid the risk of overfitting caused by using formatting prompt templates, and ensure that the text analysis system has a high degree of generalization ability and can adapt to various non-standard communication scenarios;

[0091] iv. Strengthen the mechanism for mining dangerous behaviors, pay special attention to and mark potential violence, bias and sensitive content, and improve the system's sensitivity to dangerous content;

[0092] v. Use ChatML (Chat Markup Language) format as the standard for data organization and exchange. This format integrates multiple information such as conversation context management, user interaction details, and system response design, providing a powerful tool for text analysis systems to accurately parse conversation structure, deeply explore contextual meaning, and improve the accuracy and naturalness of responses.

[0093] c.3. Model fine-tuning based on prompt words: It is an innovative prompt conversion mechanism that can restructure the input text information according to the preset template to ensure that the information is processed in an optimized form, so as to maximize the potential of the pre-trained language model and effectively extract and utilize the deep knowledge in the context. It can enable the model to learn with a small number of samples or zero samples, so that the system can quickly adapt to and solve new analysis tasks in the case of extremely limited or no additional labeled data. It focuses on adapting to a variety of downstream text analysis tasks through flexible prompt learning strategies. This process usually does not require a comprehensive update of the model parameters, thereby maintaining the original generalization ability of the model while enhancing task adaptability and efficiency and reducing dependence on additional computing resources and labeled data.

[0094] c.4. Fine-tuning of model parameters based on LoRA: Ensure that local and targeted adjustments to the model are made while maintaining the overall architecture of the model, thereby improving the performance of the model on specific tasks while retaining the generalization ability of the model; specifically, including:

[0095] i. The pre-trained model parameters are represented as θ. During fine-tuning, LoRA does not modify θ directly, but introduces a set of low-rank matrices A and S for each layer of weights W, where A is a low-rank matrix called the adaptation matrix, which is usually much smaller than W and is used to capture task-specific feature changes; S is a scaling matrix used to control the intensity of the influence of A on W and maintain an appropriate scale. The fine-tuned weight W' can be calculated by the following formula: W′=W+α·A·S; where α is an adjustable hyperparameter used to globally control the intensity of the influence of A on W. By adjusting the value of α, the weights between the original pre-trained weights and task-specific adaptation can be balanced.

[0096] ii. Low-rank approximation: The low-rank nature of A and S means that their dimensions are much smaller than W, which greatly reduces the number of parameters that need to be trained, thereby reducing computational complexity and memory usage, making fine-tuning more efficient;

[0097] iii. Only A and S matrix elements need to be optimized, which is far less than the number of original model parameters, significantly reducing computational and memory requirements during fine-tuning, which is particularly critical when dealing with specific task fine-tuning in resource-constrained environments compared to directly adjusting the entire model parameters;

[0098] iv. Through the LoRA mechanism, the model can quickly adapt to the analysis needs of different features without the need to train from scratch, which not only shortens the training cycle, but also reduces the dependence on large amounts of labeled data, thereby improving the flexibility and economy of model deployment.

[0099] Furthermore, in the inference stage of the large language model, an innovative attention mechanism calculation method is adopted, which is specially designed to accelerate the calculation of the self-attention layer in the Transformer large language model, effectively improving the computing efficiency and memory utilization. Through data segmentation, local normalization and hardware optimization strategies, GPU storage access is significantly reduced, which is particularly suitable for large-scale sequence data processing. In the specific implementation, the Q, K, and V matrices are processed in blocks, and the calculation process is optimized by using the intra-block and cross-block similarity calculations. The intra-block normalization and weighted summation are combined to efficiently generate attention weights. Through global normalization and output combination strategies, it is ensured that the model maintains the prediction performance while reducing the consumption of computing resources, and realizes efficient support for large-scale data processing on the basis of maintaining the model performance. The computational efficiency and memory utilization efficiency of the self-attention mechanism are significantly improved, which is particularly suitable for large-scale sequence data processing scenarios, while maintaining the prediction performance of the model, including:

[0100] d.1. Data partitioning: Divide the query matrix Q, key matrix K and value matrix V into T along the sequence length dimension c non-overlapping blocks, each block size is B*d, B = [L / T c ]; where L is the sequence length and d is the feature dimension;

[0101] d.2. Intra-block similarity calculation: For each block i, calculate the intra-block and cross-block similarity score matrix and Use the following formula:

[0102]

[0103] Among them, q im represents the mth query vector of the ith block in the query matrix, k in represents the nth key vector of the i-th block in the key matrix, k jn represents the nth key vector of the jth block in the key matrix, is the intra-block similarity score matrix; is the matrix of cross-block similarity scores. T represents the transpose operation, which converts the rows of the matrix into columns or converts the columns into rows so that the dot product operation can be performed.

[0104] d.3. Normalization and weighted summation within each block: Similarity matrix within each block Apply row normalization (optionally subtracting the maximum value to stabilize the calculation) to obtain the attention weight matrix P ii , P ii [m,:] represents the normalized attention weight within the block, that is, the attention distribution of the mth row in the i-th block, P ii [m,:] is calculated by the following formula:

[0105]

[0106] Then, use P ii For the corresponding value block V ii Perform weighted summation to obtain the output matrix O of the block ii ;

[0107] d.4. Global normalization and output combination: If there is cross-block similarity, merge the calculation results of all blocks and perform global normalization, and finally combine the output matrix O of all blocks ii To recover the complete output matrix O, ensuring the consistency of global attention.

[0108] Furthermore, in the inference phase, a dynamic memory management strategy is adopted to accelerate the inference process of large language models. This strategy involves dividing the input data into multiple subsequences, allocating computing resources to each subsequence in a targeted manner, and using the PagedAttention algorithm to dynamically load and unload the key and value vectors of the attention mechanism to the GPU memory. By presetting the memory page size to match the GPU cache capacity, the key-value data of the current subsequence is called in on demand while clearing the memory that is no longer necessary, thereby completing the attention calculation of the large-scale language model while ensuring efficient memory utilization, and finally summarizing the inference results of each subsequence to produce an overall analysis output. It can effectively alleviate the problem of tight memory resources in edge computing environments and improve inference efficiency. Specifically include:

[0109] e.1. Collect the input data sequence to be inferred;

[0110] e.2. dividing the input data sequence into a plurality of continuous subsequences, each subsequence corresponding to one or more steps of model processing;

[0111] e.3. Dynamically allocate computing resources for processing each subsequence, especially using the PagedAttention algorithm for memory management of the key-value pairs in the attention mechanism. The algorithm is implemented through the following sub-steps:

[0112] i. predetermine a memory page size that is suitable for loading into a graphics processor (GPU) cache at one time;

[0113] ii. The keys and values ​​required for the current processing subsequence are loaded from external storage into the current memory page of GPU memory as needed, while the memory pages occupied by the keys and values ​​of the previous subsequence that are no longer needed are released to maintain memory usage efficiency;

[0114] iii. Perform attention calculations using the keys and values ​​loaded into GPU memory to assign attention weights to the input data sequence;

[0115] e.4. Summarize the processing results of all subsequences to generate the final inference output.

[0116] The user behavior analysis module analyzes user behavior data to mark potential violence, prejudice, and sensitive content in user behavior; evaluates the severity and possible impact of these dangerous behaviors; and provides personalized suggestions or warnings to users based on user behavior reports and dangerous behavior identification. For example, if the system detects that a user frequently accesses applications with violent content, the user can be advised to reduce their use or avoid these applications altogether. Through functions such as identification, evaluation, and suggestions, the user behavior analysis module can help users identify and reduce potentially dangerous behaviors, and improve their safety and well-being on the Internet.

[0117] The above description is only a preferred embodiment of the present invention and does not constitute any other form of limitation to the present invention. Any modification or equivalent change made based on the technical essence of the present invention still falls within the scope of protection required by the present invention.

Claims

1. A user behavior analysis system based on cloud-edge-end collaboration with a large language model: comprising a device information collection module, a cloud-edge-end collaboration module and a user behavior analysis module, characterized in that: Device information collection module: By establishing a secure connection with the mobile device, it obtains hardware information and application data, and creates a local database on the edge server for data backup; it merges the collected information, encrypts it according to preset rules, generates an application database key, and implements encrypted storage and on-demand decoding access of data; Cloud-edge-end collaboration module: Full-duplex communication is achieved through Websocket and MQTT to ensure real-time information synchronization and business collaboration between cloud, edge and end, including device authentication, remote management and autonomy. User behavior analysis module: including large language models, using trained models to predict new data; Analyze information in mobile applications through large language models.

2. According to claim 1, a user behavior analysis system based on cloud-edge-end collaboration with a large language model is characterized by: The device information collection module collects data applied in the mobile device, specifically including: a.

1. Connect the device to be collected to the system, establish a link between the information collection module and the mobile device, obtain the hardware information of the device to be collected, collect application data, and generate a local database on the edge side to back up the application data; a.

2. The information acquisition module combines the hardware information and the application information, encrypts them according to the rules of the application database, obtains the key of the application database, and then decodes the database to obtain the application data.

3. According to claim 1, a user behavior analysis system based on cloud-edge-end collaboration with a large language model is characterized by: The cloud-edge-end collaboration module is used to provide information and business collaboration between the collected mobile devices and edge nodes, and between edge nodes and cloud platforms, specifically including: b.

1. Full-duplex communication based on Websocket and MQTT completes cloud-edge communication, supports synchronous communication between cloud and edge nodes based on Web applications, and obtains hardware information and application information of mobile devices in real time; supports the cloud-edge-end integrated system framework to authenticate and authorize edge devices, and implements effective management of edge devices; supports information collaboration between edge and cloud; b.

2. The edge side of the cloud-edge-device collaboration module provides edge computing business modules, which include edge container management, edge node management, mobile device access management, large language model management, and edge OS. The platform side of the cloud-edge-end collaboration module provides a cloud-edge-end collaboration console, edge device instance management, and edge device driver management; the cloud-edge-end collaboration module provides capability support for upstream applications, including user portrait generation applications and user portrait analysis applications; after the device information is collected, the cloud-edge-end collaboration module synchronizes the information privacy to the cloud for efficient analysis.

4. According to claim 1, a user behavior analysis system based on cloud-edge-end collaboration with a large language model is characterized by: The user behavior analysis module builds a large language model based on the LLaMa model that can analyze user mobile application information, specifically including: c.

1. At the network level of the large language model, the LLaMa model has been improved: c.

2. The pre-training data level includes the following: c.2.

1. Integrate multi-dimensional dialogue data annotation technology to cover multiple aspects of two-person and multi-person dialogue understanding, build comprehensive and rich training resources, and enhance the system's understanding of complex dialogue structures; c.2.

2. Use fine-grained label annotation to help the model learn language expressions and interaction strategies that are closer to human communication habits, and optimize the model's performance on various auxiliary tasks; c.2.

3. Use an exclusion mechanism to ensure that the text analysis system has a high degree of generalization ability and can adapt to various non-standard communication scenarios; c.2.

4. Strengthen the dangerous behavior mining mechanism, pay attention to and mark potential violence, prejudice and sensitive content, and improve the system's sensitivity to dangerous content; c.2.

5. Use ChatML format as the standard for data organization and exchange to accurately analyze the dialogue structure, deeply explore the contextual meaning, and improve the accuracy and naturalness of the response; c.

3. Model fine-tuning based on prompt words: structurally reorganize the input text information according to the preset template to ensure that the information is processed in an optimized form; c.

4. Fine-tuning of model parameters based on LoRA: Ensure local and targeted adjustments to the model while maintaining the overall architecture of the model, thereby improving the performance of the model on specific tasks while retaining the generalization ability of the model.

5. According to claim 4, a user behavior analysis system based on cloud-edge-end collaboration with a large language model is characterized by: At the network level of the large language model described in c.1, the LLaMa model has been improved including the following: c.1.

1. Use a pre-normalization strategy to apply normalization processing before each layer of the neural network to improve the stability of the training process and the convergence efficiency of the model; c.1.

2. Use RMSNorm as the normalization method. RMSNorm calculates the root mean square value of each dimension of the input tensor and scales it accordingly, thus providing enhanced stability when processing input data with different scales. c.1.

3. Select SwiGLU as the activation function. The SwiGLU activation function combines the characteristics of the Swish function and the gated linear unit; c.1.

4. Adjust the dimensionality of the feedforward network, reducing it from four times the traditional hidden layer dimension to three-eighths of the hidden dimension, optimizing the model's storage requirements and computing resource utilization efficiency while maintaining or improving the model's learning ability and expressiveness.

6. The user behavior analysis system based on cloud-edge-end collaboration with a large language model according to claim 4 is characterized by: The LoRA-based model parameter fine-tuning described in c.4 specifically includes: c.4.

1. The pre-trained model parameters are represented as θ. During fine-tuning, LoRA does not modify θ directly, but introduces a set of low-rank matrices A and S for each layer weight W, where A is a low-rank matrix whose size is much smaller than W and is used to capture task-specific feature changes; S is a scaling matrix used to control the intensity of the influence of A on W and maintain an appropriate scale; the fine-tuned weight W' is calculated by the following formula: W′=W+α·A·S; where α is an adjustable hyperparameter used to globally control the intensity of the influence of A on W; by adjusting the value of α, the weight between the original pre-trained weight and the task-specific adaptation is balanced; c.4.

2. Low-rank approximation: The low-rank nature of A and S means that their dimensions are much smaller than W, which greatly reduces the number of parameters that need to be trained, thereby reducing computational complexity and memory usage, making fine-tuning more efficient; c.4.

3. The number of A and S matrix elements to be optimized is far less than the number of original model parameters, which significantly reduces the computational and memory requirements during fine-tuning.

7. The user behavior analysis system based on cloud-edge-end collaboration with a large language model according to claim 1 is characterized by: The user behavior analysis module uses the attention mechanism calculation method to accelerate the reasoning stage, which specifically includes the following contents: d.

1. Data partitioning: Divide the query matrix Q, key matrix K and value matrix V into T along the sequence length dimension c non-overlapping blocks, each block size is B*d, B = [L / T c ]; where L is the sequence length and d is the feature dimension; d.

2. Intra-block similarity calculation: For each block i, calculate the intra-block and cross-block similarity score matrix and , using the following formula: Among them, q im represents the mth query vector of the ith block in the query matrix, k in represents the nth key vector of the i-th block in the key matrix, k jn represents the nth key vector of the jth block in the key matrix, is the intra-block similarity score matrix; is the cross-block similarity score matrix; T represents the transpose operation, which converts the rows of the matrix into columns or converts the columns into rows so that the dot product operation can be performed; d.

3. Normalization and weighted summation within each block: Similarity matrix within each block Apply row normalization to get the attention weight matrix P ii , P ii [m,:] represents the normalized attention weight within the block, that is, the attention distribution of the mth row in the i-th block, P ii [m,:] is calculated by the following formula: Then, use P ii For the corresponding value block V ii Perform weighted summation to obtain the output matrix O of the block ii ; d.

4. Global normalization and output combination: If there is cross-block similarity, merge the calculation results of all blocks and perform global normalization, and finally combine the output matrix O of all blocks ii To recover the complete output matrix O, ensuring the consistency of global attention.

8. The user behavior analysis system based on cloud-edge-end collaboration with a large language model according to claim 7 is characterized by: The attention mechanism calculation method is used to accelerate reasoning, and the dynamic memory management strategy is used to accelerate the reasoning process of large language models. The input data is divided into multiple subsequences, each subsequence is targeted to allocate computing resources, and the PagedAttention algorithm is used to dynamically load and unload the key and value vectors of the attention mechanism to the GPU memory. By presetting the memory page size to match the GPU cache capacity, the key-value data of the current subsequence is called in as needed while clearing the memory that is no longer necessary, thereby completing the attention calculation of the large-scale language model while ensuring efficient memory utilization, and finally summarizing the reasoning results of each subsequence to produce an overall analysis output.

9. The user behavior analysis system based on cloud-edge-end collaboration with a large language model according to claim 8, characterized in that: A dynamic memory management strategy is used to speed up the reasoning process of large language models, including the following: e.

1. Collect the input data sequence to be inferred; e.

2. dividing the input data sequence into a plurality of continuous subsequences, each subsequence corresponding to one or more steps of model processing; e.

3. Dynamically allocate computing resources for the processing of each subsequence, and use the PagedAttention algorithm for memory management for the key-value pairs keys and values ​​in the attention mechanism.

10. The user behavior analysis system based on cloud-edge-end collaboration with a large language model according to claim 9 is characterized by: The PagedAttention algorithm is used for memory management as described in e.3, including the following: e.3.

1. Predetermine a memory page size that is suitable for loading into the GPU cache at one time; e.3.

2. The keys and values ​​required for the current processing subsequence are loaded from external storage into the current memory page of GPU memory as needed, while the memory pages occupied by the keys and values ​​of the previous subsequence that are no longer needed are released to maintain memory usage efficiency; e.3.

3. Perform attention calculations using the keys and values ​​loaded into GPU memory to assign attention weights to the input data sequence. e.3.

4. Summarize the processing results of all subsequences to generate the final inference output.

Images