Artificial intelligence-based distributed collaborative office data processing method

Abstract

This invention discloses an artificial intelligence-based distributed collaborative office data processing method, belonging to the field of collaborative office data processing technology. The method includes acquiring office data resources and performing data lake layering and merging to form cloud-native data assets, and extracting high-dimensional feature vectors to construct a feature engineering set; inputting the feature engineering set into a federated learning model for incremental training to generate a global inference model; using the global inference model to identify abnormal behavior patterns and associating them with original log data to generate abnormal event snapshots; constructing a causal graph from the abnormal event snapshots, and mapping abnormal types with business entities through the knowledge graph to form a root cause analysis chain; automatically triggering the data governance workflow to recalculate statistical indicators, and updating the feature engineering set through secondary feature fusion and feeding it back to the model for the next round of training. This method can achieve accurate tracing of abnormal root causes and a closed loop of data governance and model iteration, improving the effectiveness of collaborative office data processing.

Description

Technical Field

[0001] This invention belongs to the field of collaborative office data processing technology, specifically a distributed collaborative office data processing method based on artificial intelligence. Background Technology

[0002] Currently, collaborative office data processing often combines artificial intelligence and cloud computing technologies. It involves merging and organizing office data resources through a data lake layering system, constructing a feature engineering set, inputting it into a machine learning model for training, and then using the trained model to identify data anomalies, thereby conducting data governance related work. Federated learning technology has also been applied to collaborative office data processing to achieve collaborative training of multi-source heterogeneous data, avoid data privacy leaks, and improve model adaptability.

[0003] In existing technologies, after a model identifies anomaly patterns, it struggles to establish an effective correlation with the original log data, making it impossible to accurately trace the original source of the anomaly. The root cause analysis process lacks deep integration with business entities, resulting in poor interpretability of the root cause analysis results and an inability to clearly define the relationship between the anomaly and specific business processes. When building the data foundation for AI analysis, existing methods typically use general-purpose data warehouses or single-structure data lakes for storage. They lack cloud-native data asset construction methods that are deeply coupled with subsequent federated learning and anomaly analysis for multi-source heterogeneous data in collaborative office scenarios, such as unstructured logs, semi-structured business process data, and structured statistical indicators. Feature engineering often focuses on simple processing of single-type data, failing to deeply integrate and represent data at intrinsic levels such as the original semantics of the source layer, the business attributes of the standard layer, and the statistical trends of the aggregation layer. The constructed feature sets are insufficient to comprehensively and accurately depict the complex behavioral patterns in collaborative office work, limiting the effectiveness of subsequent model training. Meanwhile, data governance work often relies on manual triggering, and the results of data quality recalculation after governance cannot be fed back to the model training process in a timely manner. This results in the feature engineering set not being able to be dynamically updated, the incremental training of federated learning models lacks continuous data support, makes it difficult to achieve dynamic optimization of model adaptability, and fails to form a complete closed loop of data processing, anomaly identification, root cause analysis, data governance and model iteration in collaborative office scenarios. Summary of the Invention

[0004] This invention aims to solve at least one of the technical problems existing in the prior art; Therefore, this invention proposes a distributed collaborative office data processing method based on artificial intelligence, including: Acquire office data resources to be processed, perform data lake layering and merging on the office data resources to form cloud-native data assets, extract high-dimensional feature vectors from them, and construct feature engineering sets; The feature engineering set is input into a pre-built federated learning model for incremental training to generate a global inference model that adapts to the distribution of multi-source heterogeneous data. The global inference model is used to identify abnormal behavior patterns of the cloud-native data assets. The identified abnormal behavior patterns are then associated with the original log data in the data lake source layer to generate an abnormal event snapshot that includes the anomaly type, scope of impact, and time of occurrence. A causal graph is constructed on the snapshot of the abnormal events. The abnormal types are mapped to business entities in the standard layer of the data lake using knowledge graph technology to form a highly interpretable root cause analysis chain. Based on the root cause analysis chain, a data governance workflow is automatically triggered in the cloud computing platform. The data governance workflow recalculates the data quality of relevant statistical indicators in the data lake aggregation layer according to the business entities involved in the root cause analysis chain. The recalculated statistical indicators are then fused with the business data in the standard layer of the data lake to update the feature engineering set, which is then input into the federated learning model for the next round of training.

[0005] Furthermore, the process of forming cloud-native data assets and extracting high-dimensional feature vectors from them to construct a feature engineering set includes: The office data resources to be processed are obtained through the cloud storage system. The office data resources include unstructured log data, semi-structured business data, and structured indicator data. The office data resources are layered and merged into a data lake. Unstructured log data, semi-structured business data, and structured indicator data are imported into the source layer, standard layer, and aggregation layer of the data lake, respectively, to form cloud-native data assets with unified timestamps and metadata tags. High-dimensional feature vectors are extracted from the cloud-native data assets. These high-dimensional feature vectors are composed of business data attributes in the data lake standard layer, text semantic features in the source layer, and statistical indicators in the aggregation layer, in order to construct a feature engineering set for model training.

[0006] Furthermore, the office data resources are layered and merged within the data lake, with unstructured log data, semi-structured business data, and structured indicator data imported into the source layer, standard layer, and aggregation layer of the data lake, respectively, including: Perform a row-delimiter-based splitting operation on the unstructured log data, convert the split log entries into key-value pair format with source system tags, and store them in the source-attached layer of the data lake; The pattern matching algorithm is applied to the semi-structured business data to compare the field names in the semi-structured business data with the predefined data dictionary of the data lake standard layer. Missing fields are filled in and redundant fields are removed to form a structured data table that conforms to the data lake standard layer specification. Time window aggregation is performed on the structured indicator data to uniformly convert indicator data with different time granularities to a statistical caliber based on hours, and data quality check bits are added to generate a summary data table in the data lake aggregation layer. Extracting high-dimensional feature vectors from the cloud-native data assets includes: One-hot encoding is performed on the business data attributes in the standard layer of the data lake to convert categorical variables into numerical sparse vectors; Word embedding technology is applied to the text semantic features in the data lake source layer to convert the text data into a dense vector of fixed dimensions; Max-min normalization is performed on the statistical indicators in the data lake aggregation layer to map indicator data of different dimensions to the same value range. The one-hot encoding results, word embedding results, and normalization results are concatenated along the feature dimension to form the high-dimensional feature vector.

[0007] Furthermore, the feature-engineered set is input into a pre-built federated learning model for incremental training, including: The federated learning model maintains only global model parameters in the cloud and local training data on edge computing nodes. The network structure and hyperparameters of the federated learning model are initialized in the cloud, and the network structure and hyperparameters are distributed to each edge computing node participating in the training. Each edge computing node uses local data lake standard layer data to perform several rounds of forward and backward propagation on the federated learning model locally in order to calculate the update amount of local model parameters. Each edge computing node encrypts the update amount of its local model parameters and uploads it to the cloud. The cloud performs a weighted average of the update amounts from all edge computing nodes to update the global model parameters. The updated global model parameters are then distributed to each edge computing node to begin the next round of incremental training.

[0008] Furthermore, the global inference model is used to identify abnormal behavior patterns in the cloud-native data assets, and the identified abnormal behavior patterns are correlated with the raw log data in the data lake's source layer, including: The high-dimensional feature vectors in cloud-native data assets are input into the global inference model, which outputs the probability value of each data sample belonging to normal or abnormal behavior. Samples with probability values ​​exceeding the anomaly threshold are filtered out, and their unique identifiers in the data lake are recorded. Based on the unique identifier, the corresponding original log data is retrieved from the data lake post source layer, and the operation user, operation object and operation time information in the original log data are extracted to generate the abnormal event snapshot.

[0009] Furthermore, a causal graph is constructed from the snapshots of the abnormal events, and knowledge graph technology is used to map the relationship between the abnormal types and business entities in the standard layer of the data lake, including: Using the exception type in the exception event snapshot as the head entity and the scope of impact involved in the exception event snapshot as the tail entity, establish a "cause" relationship edge in the graph database; Using business entities in the standard layer of the data lake as nodes and business dependencies between business entities as edges, construct a business entity relationship subgraph; The "cause" relationship edge is merged with the business entity relationship subgraph, and a path search is performed in the graph database to find the shortest propagation path from the anomaly type to the specific business entity. The shortest propagation path is the root cause analysis chain.

[0010] Furthermore, based on the root cause analysis chain, a data governance workflow is automatically triggered in the cloud computing platform, including: Analyze the business entities involved in the root cause analysis chain to determine the list of statistical indicators that need to be recalculated; A data quality recalculation task is created on the cloud computing platform. The data quality recalculation task reads the raw data of relevant statistical indicators in the data lake aggregation layer and cleans and repairs the raw data according to data quality rules. The cleaned and repaired data is recalculated to generate recalculated statistical indicators, which then overwrite the old indicator data in the data lake aggregation layer.

[0011] Furthermore, the step of performing secondary feature fusion between the recalculated statistical indicators and the business data in the data lake standard layer to update the feature engineering set includes: The recalculated statistical indicators are joined with the business data in the standard layer of the data lake in the horizontal dimension to form a wide table containing historical and new features. Perform Pearson correlation analysis on the new features in the wide table to remove redundant new features that are highly correlated with existing features; The new features, after being filtered, are concatenated with the original high-dimensional feature vectors to form an updated feature engineering set.

[0012] Furthermore, it also includes: During training, the computing resource consumption of the federated learning model on edge computing nodes is collected through cloud monitoring services. When the computing resource consumption exceeds the preset elastic scaling threshold, the number of container instances on the cloud platform is dynamically adjusted to allocate corresponding computing resources. When the federated learning model converges over multiple consecutive training cycles, the final global inference model is published as an online prediction service in the cloud to complete the intelligent processing of big data. During training, the cloud monitoring service collects data on the computing resource consumption of the federated learning model on edge computing nodes. When the computing resource consumption exceeds a preset elastic scaling threshold, the number of container instances on the cloud platform is dynamically adjusted, including: The cloud monitoring service periodically collects CPU utilization, memory usage, and network bandwidth usage on each edge computing node through an agent program. When the CPU utilization or memory usage exceeds the elastic scaling threshold, a scaling request is sent to the scheduling center of the cloud computing platform. The scheduling center then starts a new container instance in the cloud and migrates some computing tasks to the new container instance. When both CPU utilization and memory utilization are below the scaling-down threshold, a scaling-down request is sent to the scheduling center, which then stops some idle container instances to release computing resources.

[0013] Furthermore, when the federated learning model converges over multiple consecutive training epochs, the final global inference model is published as an online prediction service in the cloud, including: After the federated learning model is trained, the global inference model is serialized, and the model's structure and parameters are saved as a model file in a specific format. Upload the model files to the cloud-based model repository, and configure the version number and access interface for the model files in the model repository; A prediction service instance is deployed in a container orchestration platform in the cloud. The prediction service instance loads model files from the model repository and provides a prediction interface based on the Hypertext Transfer Protocol to complete the intelligent processing of big data.

[0014] Compared with the prior art, the beneficial effects of the present invention are: After identifying anomalous behavior patterns in cloud-native data assets using a global inference model, the identified anomalous behavior patterns are directly correlated with the raw log data in the data lake's source layer. This generates a snapshot of anomalous events, including the anomaly type, scope of impact, and time of occurrence. A causal graph is then constructed from this snapshot, and knowledge graph technology is used to accurately map the anomaly type to business entities in the data lake's standard layer, forming a highly interpretable root cause analysis chain. This solution enables precise correlation between anomalous events and raw log data, clearly identifying the original data source of the anomaly. Through knowledge graphs, it achieves deep binding between anomalies and business entities, making the root cause analysis process clear and traceable. This overcomes the limitations of conventional anomaly identification being disconnected from raw data and root cause analysis lacking business relevance, making the location of anomaly root causes more targeted and preventing superficial root cause analysis.

[0015] Based on the root cause analysis chain, a data governance workflow is automatically triggered in the cloud computing platform. This workflow recalculates the data quality of relevant statistical indicators in the data lake aggregation layer according to the business entities involved in the root cause analysis chain. The recalculated statistical indicators are then fused with business data in the data lake standard layer to update the feature engineering set. The updated feature engineering set is then re-input into the federated learning model to complete the next round of incremental training. This solution automates the triggering of data governance without manual intervention, reducing manual operation costs and ensuring that data governance accurately corresponds to the business links involved in the root causes of anomalies. At the same time, it allows high-quality data after data governance to be fed back into model training in a timely manner, forming a closed-loop iteration between data governance and model training. This solves the problem of the disconnect between conventional model training and data governance, enabling the federated learning model to continuously adapt to changes in the distribution of multi-source heterogeneous data, improving the consistency and stability of data processing. Attached Figure Description

[0016] Figure 1 This is a flowchart illustrating the steps of the distributed collaborative office data processing method based on artificial intelligence described in this invention. Figure 2 A flowchart for creating cloud-native data assets and extracting features; Figure 3 The loss curve during the training process of the federated learning model; Figure 4 A flowchart for identifying and associating abnormal behaviors; Figure 5 This is a monitoring curve for resource consumption of edge computing nodes. Detailed Implementation

[0017] The technical solution of the present invention will be clearly and completely described below with reference to the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] See Figure 1 This paper describes an AI-based distributed collaborative office data processing method. The method acquires office data resources to be processed, performs data lake layering and merging on these resources to form cloud-native data assets, extracts high-dimensional feature vectors from them, constructs a feature engineering set, and inputs this feature engineering set into a pre-built federated learning model for incremental training. This generates a global inference model adapted to multi-source heterogeneous data distribution. The global inference model is then used to identify abnormal behavior patterns in the cloud-native data assets. The identified abnormal behavior patterns are correlated with the original log data in the data lake's source layer to generate a data structure including the anomaly type, scope of impact, and occurrence time. Anomaly event snapshots are generated, and a causal graph is constructed from these snapshots. Knowledge graph technology is used to map anomaly types to business entities in the data lake's standard layer, forming a highly interpretable root cause analysis chain. Based on this chain, a data governance workflow is automatically triggered in the cloud computing platform. This workflow recalculates the data quality of relevant statistical indicators in the data lake's aggregation layer according to the business entities involved in the root cause analysis chain. The recalculated statistical indicators are then fused with the business data in the data lake's standard layer to update the feature engineering set, which is then input again into the federated learning model for the next round of training.

[0019] In one embodiment of the present invention, see [reference] Figure 2The system acquires office data resources to be processed through a cloud storage system. These resources include unstructured log data, semi-structured business data, and structured indicator data. In specific implementations, unstructured log data refers to the raw operation log files generated by the office system, semi-structured business data is JSON-formatted cross-departmental collaborative task records, and structured indicator data is statistical tables in the database, such as task completion rate and online meeting duration. The office data resources are then layered and merged within a data lake. The unstructured log data, semi-structured business data, and structured indicator data are imported into the source layer, standard layer, and aggregation layer of the data lake, respectively, to form cloud-native data assets with unified timestamps and metadata tags. In some embodiments, the unstructured log data undergoes a line-delimited segmentation operation. For example, for log entries separated by newline characters, each segmented log entry is converted into a key-value pair format with the tag `source_system:oa_system` or `source_system:im_system`, and stored in the source layer of the data lake. Optionally, a pattern matching algorithm is applied to the semi-structured business data to compare the field names in the semi-structured business data with the predefined data dictionary of the data lake standard layer. For example, when the predefined data dictionary contains fields such as task_id, assignee_department, status, and deadline, and the input JSON format collaborative task record contains fields such as task_identifier and assignee, the task_identifier field is renamed to task_id, the assignee field is mapped to assignee_department, the missing deadline field is filled in, and the redundant internal_notes field is removed to form a structured data table that conforms to the data lake standard layer specification. In practice, time window aggregation is performed on the structured indicator data. For example, the indicator data such as 'task completion rate' and 'online meeting duration' in the original database, which are granular in minutes, hours, and days, are uniformly converted to a statistical caliber based on days by rounding up to the nearest day. Data quality check bits that indicate data integrity are added to the aggregated data for each day to generate a summary data table in the data lake aggregation layer.

[0020] High-dimensional feature vectors are extracted from the cloud-native data assets. These vectors consist of business data attributes from the standard layer of the data lake, textual semantic features from the source layer, and statistical indicators from the aggregation layer, forming a feature engineering set for model training. The steps of constructing cloud-native data assets and extracting high-dimensional feature vectors are specifically designed for the multi-source, heterogeneous, and hierarchical characteristics of data in distributed collaborative office scenarios, and are not general data storage or feature engineering methods. Specifically, unstructured logs, semi-structured business data, and structured indicator data are respectively categorized into the source layer, standard layer, and aggregation layer of the data lake. This hierarchical strategy aims to preserve original data details, standardize business entities, and accumulate statistical knowledge, thereby forming a data asset system with clear internal logic that facilitates subsequent in-depth analysis. Based on this, features such as textual semantics, business attributes, and statistical indicators are extracted and integrated from these three different layers. This ensures that the final high-dimensional feature vectors can simultaneously cover the user's original intent, business context, and macro-level behavioral patterns, providing the subsequent federated learning model with input that comprehensively and accurately represents the complex behavioral patterns of collaborative office work. This method, which is deeply coupled with business scenarios and designed and integrated in a targeted manner based on the inherent hierarchy of data, is an important foundation for achieving accurate anomaly identification and root cause analysis. In some embodiments, the business data attributes in the standard layer of the data lake are one-hot encoded, and the values ​​of the categorical variable 'task priority' ('high', 'medium', 'low') are converted into numerical sparse vectors [1,0,0], [0,1,0], [0,0,1]. Optionally, word embedding technology is applied to the text semantic features in the source layer of the data lake. For example, through a pre-trained Word2Vec model, the text data DELETE / api / v1 / document / 1001 or ACCESS / meeting / room / book in the log entries are converted into dense vectors of fixed dimensions [0.12,-0.45,...,0.78]. The statistical indicators in the data lake aggregation layer undergo max-min normalization. For example, for the statistical indicator "number of online users," whose original value range is [100, 10000], the normalization result for a value of 5000 is calculated as (5000-100) / (10000-100) = 0.495, thus mapping indicator data of different dimensions to the [0,1] interval. In specific implementation, the normalization process can be expressed as: for the i-th statistical indicator value... Its normalization result The calculation formula is:

[0021] in: It is the normalized result of the i-th indicator. It is the original value of the i-th indicator. It is the set of all values ​​corresponding to the j-th indicator. It is a function that takes the minimum value in a set. It is a function that takes the maximum value in the set. It can be understood that the one-hot encoding result, word embedding result and normalization result are concatenated in the feature dimension. For example, the one-hot encoding vector with dimension 3, the word embedding vector with dimension 100 and the normalization index vector with dimension 5 are concatenated end to end to form the high-dimensional feature vector with dimension 108.

[0022] In one embodiment of the present invention, the federated learning model maintains only global model parameters in the cloud and local training data on edge computing nodes. In a specific implementation, the cloud can be a centralized model parameter server, and the three edge computing nodes can be business servers located in different geographical regions. Each edge computing node locally stores data lake standard layer data containing user behavior data tables for its corresponding region. The network structure and hyperparameters of the federated learning model are initialized in the cloud. The network structure can be a fully connected neural network containing an input layer, hidden layers, and an output layer. The hyperparameters include the learning rate, training batch size, and local training epochs. The network structure and hyperparameters are then distributed to each edge computing node participating in the training. In some embodiments, the distributed learning rate is set to 0.01, the training batch size is set to 32, and the local training epochs are set to 5.

[0023] Each edge computing node utilizes its local data lake standard layer data to perform several rounds of forward and backward propagation on the federated learning model locally to calculate the update amount of the local model parameters. For example, edge computing node A uses 10,000 local user transaction data points. In 5 rounds, it calculates the loss between the model's predicted value and the true label in each forward propagation, then calculates the gradient of the loss function with respect to the model parameters using the backward propagation algorithm, and updates the local model parameters using gradient descent. Finally, it calculates the difference between the model parameters at the beginning and the end of the current training round to obtain the update amount of the local model parameters. Each edge computing node encrypts the update amount of the local model parameters and uploads it to the cloud. The encryption method can be homomorphic encryption to ensure the privacy of the parameters during transmission and aggregation. Optionally, the cloud performs a weighted average of the update amounts from all edge computing nodes to update the global model parameters. The weight of the weighted average can be determined based on the proportion of each node's local data to the total data volume. In specific implementation, the process of updating the global model parameters can be expressed by the following formula:

[0024] in: This represents the updated global model parameters for round t+1. This represents the global model parameters at the start of the t-th training round. It is the total number of federated learning edge computing nodes participating in the training. It is the weight coefficient of the k-th edge computing node. This represents the encrypted local model parameter update amount uploaded by the k-th edge computing node. This can be understood as redistributing the updated global model parameters to each edge computing node, for example... The parameter matrix is ​​distributed to edge computing nodes A, B, and C to begin the next round of incremental training.

[0025] See Figure 3 In the incremental training process of the federated learning model, the loss change curves of the global model and edge nodes intuitively reflect the convergence characteristics of distributed training under multi-source heterogeneous data. Specifically, in the early stage of training, the loss values ​​of each node decrease rapidly, and the loss of the global model and the edge node maintain a synchronous convergence trend, reflecting the effective transmission of local gradient updates in global parameter aggregation. As training progresses, the rate of loss decrease slows down. Due to the difference in local data distribution, the loss value of edge node 3 is always slightly higher than that of other nodes and the global model, reflecting the robust adaptation of federated learning to heterogeneous data distribution. In the later stage of training, the losses of the global model and each edge node stabilize in the range of 0.32-0.38, and the curves tend to flatten, indicating that the model has reached the convergence state, and the global inference ability has achieved an optimal balance under the collaboration of multi-source data. This curve fully depicts the evolution law of loss in the entire cycle of federated learning from initialization to convergence, verifies the synergistic effectiveness of global aggregation and local updates in distributed training, and provides a quantitative basis for model hyperparameter tuning and convergence determination.

[0026] In one embodiment of the present invention, see [reference] Figure 4The high-dimensional feature vectors from cloud-native data assets are input into a global inference model. The global inference model outputs a probability value for each data sample indicating whether it represents normal or abnormal behavior. For example, the global inference model is a binary classification neural network; for an input high-dimensional feature vector with dimension 108, it outputs a scalar value between 0 and 1, where 0 represents normal and 1 represents abnormal. A specific data sample might have a probability value of 0.95. Samples with probability values ​​exceeding an abnormal threshold (which can be preset to 0.9) are filtered out. The unique identifier of each sample in the data lake is recorded; this unique identifier could be "log_id:12345" corresponding to each original log data in the data lake's source layer. Based on the unique identifier, the corresponding original log data is retrieved from the data lake's post source layer. For example, based on 'log_id:12345', an original log entry '2023-10-26 14:30:25,user:zhangsan,action:DOWNLOAD,target: / confidential / projectX / design.docx,status:200' is retrieved. The operation user, operation object, and operation time information are extracted from the original log data to generate the abnormal event snapshot. The generated snapshot content is 'Abnormal type: unauthorized file download, operation user: zhangsan, operation object: / confidential / projectX / design.docx, occurrence time: 2023-10-26 14:30:25'. In some embodiments, the abnormal event snapshot may also include a confidence score calculated from probability values.

[0027] Using the exception type in the exception event snapshot as the head entity and the scope of impact involved in the exception event snapshot as the tail entity, "cause" relationship edges are established in the graph database. For example, the head entity "High-risk deletion operation" and the tail entity "User service module exception" are connected by a directed edge labeled "cause". Using business entities in the data lake's standard layer as nodes (business entities can be "department table", "project table", "document permission table"), and business dependencies between business entities as edges (e.g., "ownership" relationship between "department table" and "project table", and "association" relationship between "project table" and "document permission table"), a business entity relationship subgraph is constructed. This can be understood as merging the "cause" relationship edges with the business entity relationship subgraph and performing a path search in the graph database to find the shortest propagation path from the exception type to the specific business entity. In specific implementations, the path search can use Dijkstra's algorithm, and the shortest propagation path is the root cause analysis chain. An example of a root cause analysis chain found is "Unauthorized file download" -> (caused) -> "Document security module alert" -> (affected) -> "Project X document library" -> (belongs to) -> "R&D Department". This chain describes the propagation path of the abnormal operation. In some embodiments, the termination condition for path search can be set to when a leaf node business entity with no subsequent business dependencies is found. The path search process can be described by the following formula, used to evaluate the weight of the path:

[0028] in: This represents the overall weight of a path from the starting entity to the target entity. This represents the total number of relation edges traversed by the path. Indicates the first [item] on this path The historical association strength score between the two business entities connected by the edge. This historical association strength score can be statistically derived from historical operation and maintenance data. The path with the smallest value is taken as the shortest propagation path.

[0029] In one embodiment of the present invention, the business entities involved in the root cause analysis chain are analyzed to determine the list of statistical indicators that need to be recalculated. For example, from a root cause analysis chain "unauthorized file download -> document security module alarm -> project X document library", the business entities "project table" and "department table" are parsed out, and then the list of statistical indicators related to such business entities is determined to be "average daily access volume of project documents" and "number of completed departmental collaborative tasks". Furthermore, the list of statistical indicators related to such business entities is determined to be "daily active users" and "total number of daily orders". A data quality recalculation task is created on the cloud computing platform. The data quality recalculation task reads the original data of relevant statistical indicators in the data lake aggregation layer and cleans and repairs the original data according to data quality rules. Data quality rules may include filling in missing values ​​forward and replacing outliers exceeding three standard deviations with the sliding window mean. The cleaned and repaired data is recalculated statistically, for example, the sum and average of the repaired "daily active users" sequence are recalculated by day to generate recalculated statistical indicators, which overwrite the original old indicator data in the data lake aggregation layer.

[0030] The recalculated statistical indicators are then fused with the business data in the standard layer of the data lake to update the feature engineering set. A horizontal table join is then established between the recalculated statistical indicators and the business data in the standard layer of the data lake. Optionally, refer to Table 1 for an example of the joined wide table.

[0031] Table 1: Example Fragments of Wide Feature Fusion 10001 2023-10-26 192.168.1.1 10000 10002 2023-10-26 192.168.1.2 10000 10003 2023-10-25 192.168.1.3 10100 Perform Pearson correlation analysis on new features in the wide table to remove redundant new features that are highly correlated with existing features. For example, calculate the Pearson correlation coefficient between the new feature "active_user_count" and the existing historical feature "last_login_interval_days". In some embodiments, calculate two feature vectors. and Pearson correlation coefficient between The formula is:

[0032] in: Representation of features With features The Pearson correlation coefficient between them Indicates the sample size, and Representing features respectively and characteristics In the The values ​​on each sample and Representing features respectively and characteristics The average value across all samples. The filtered new features are then concatenated with the original high-dimensional feature vector. For example, if the original high-dimensional feature vector is 108 dimensions, and correlation analysis retains two new features, "daily_order_total" and "payment_success_rate", then the values ​​of these two new features are concatenated to the end of the original vector to form an updated 110-dimensional feature engineering set. In some embodiments, if all new features are removed due to high correlation, the feature engineering set remains unchanged.

[0033] In one embodiment of the present invention, during the training process, the computing resource consumption of the federated learning model on edge computing nodes is collected through a cloud monitoring service. The cloud monitoring service periodically collects the CPU utilization, memory usage, and network bandwidth utilization of each edge computing node through an agent program. For example, the agent program collects the CPU utilization, memory usage, and network bandwidth utilization of edge computing node A every 30 seconds, obtaining a set of data: CPU utilization 72%, memory usage 65%, and network bandwidth utilization 40%. When the computing resource consumption exceeds a preset elastic scaling threshold, the number of container instances on the cloud platform is dynamically adjusted to allocate corresponding computing resources. In some embodiments, the preset CPU utilization elastic scaling threshold is 85%, and the preset memory usage elastic scaling threshold is 80%. When the CPU utilization or memory usage exceeds the elastic scaling threshold, a scaling request is sent to the cloud computing platform's scheduling center. For example, if the CPU utilization of edge computing node A reaches 90% for three consecutive data collection cycles, exceeding the 85% threshold, the cloud monitoring service sends a scaling request to the scheduling center containing the node identifier and the required resource specifications. The scheduling center then launches a new container instance in the cloud and migrates some computing tasks to the new container instance, for example, migrating 30% of the local training tasks from edge computing node A to the newly launched container instance B. Optionally, when both CPU utilization and memory usage are below the scaling-down threshold, a scaling-down request is sent to the scheduling center. For example, if the CPU utilization of edge computing node A drops to 20% and the memory usage drops to 15% for five consecutive data collection cycles, the cloud monitoring service sends a scaling-down request to the scheduling center. The scheduling center then stops some idle container instances to release computing resources, for example, stopping container instance B and returning all computing tasks to edge computing node A for processing. It is understandable that the judgment of dynamic adjustment can be based on a comprehensive score, and the resource consumption can be quantified into an expansion trigger value using the following formula. : in: Indicates the expansion trigger value, This indicates the collected central processing unit utilization rate. This indicates the collected memory usage rate. This indicates the collected network bandwidth utilization rate. , , These are the preset weighting coefficients corresponding to CPU utilization, memory usage, and network bandwidth utilization. When the value exceeds the preset expansion threshold, an expansion operation is triggered.

[0034] See Figure 5 In the monitoring of edge computing node resource consumption during federated learning model training, the time-series curves of CPU utilization and memory usage intuitively reflect the load fluctuations and elastic scaling effects of distributed training. Specifically, in the first 20 minutes of monitoring, both CPU and memory utilization remained in the low-load range of 30%-60%, with gentle fluctuations, corresponding to the initialization and lightweight local update phases of model training. In the 20-40 minute range, CPU utilization repeatedly exceeded the preset 85% expansion threshold, and memory utilization simultaneously climbed to over 80%. This phase is the core aggregation training period of federated learning, where multi-node gradient uploads and global parameter updates caused peak resource loads, triggering the cloud platform's elastic scaling mechanism and adding container instances to offload computing tasks. After 40 minutes, resource utilization gradually dropped back to the 40%-70% range, with a significant decrease in peak frequency, reflecting the load balancing effect after scaling up and providing a basis for the scaling-down operation after model convergence. The red and yellow dashed lines in the figure mark the elastic scaling thresholds for CPU and memory, respectively, and the shaded area marks the expansion window. This fully depicts the dynamic resource scheduling process throughout the entire training cycle of federated learning and verifies the adaptability of elastic scaling strategies to distributed training loads in a cloud-native environment.

[0035] When the federated learning model converges over multiple consecutive training epochs, the final global inference model is published as an online prediction service in the cloud to enable intelligent processing of big data. For example, convergence is determined when the rate of change of the global model's loss function value is less than 0.1% over 10 consecutive training epochs. After the federated learning model training is completed, the global inference model is serialized, saving its structure and parameters as model files in a specific format, such as saving the trained neural network model structure and weight parameters as a Protobuf-formatted model.pb file. The model file is uploaded to a cloud-based model repository, where a version number and access interface are configured. A prediction service instance is deployed in a cloud-based container orchestration platform. This prediction service instance loads the model files from the model repository and provides a prediction interface based on the Hypertext Transfer Protocol (HTTP) to enable intelligent processing of big data.

[0036] The above embodiments are only used to illustrate the technical methods of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical methods of the present invention without departing from the spirit and scope of the technical methods of the present invention.

Claims

1. A distributed collaborative office data processing method based on artificial intelligence, characterized in that, include: Acquire office data resources to be processed, perform data lake layering and merging on the office data resources to form cloud-native data assets, extract high-dimensional feature vectors from them, and construct feature engineering sets; The feature engineering set is input into a pre-built federated learning model for incremental training to generate a global inference model that adapts to the distribution of multi-source heterogeneous data. The global inference model is used to identify abnormal behavior patterns of the cloud-native data assets. The identified abnormal behavior patterns are then associated with the original log data in the data lake source layer to generate an abnormal event snapshot that includes the anomaly type, scope of impact, and time of occurrence. A causal graph is constructed on the snapshot of the abnormal events. The abnormal types are mapped to business entities in the standard layer of the data lake using knowledge graph technology to form a highly interpretable root cause analysis chain. Based on the root cause analysis chain, a data governance workflow is automatically triggered in the cloud computing platform. The data governance workflow recalculates the data quality of relevant statistical indicators in the data lake aggregation layer according to the business entities involved in the root cause analysis chain. The recalculated statistical indicators are then fused with the business data in the standard layer of the data lake to update the feature engineering set, which is then input into the federated learning model for the next round of training.

2. The distributed collaborative office data processing method based on artificial intelligence according to claim 1, characterized in that, The process of forming cloud-native data assets and extracting high-dimensional feature vectors from them to construct a feature engineering set includes: The office data resources to be processed are obtained through the cloud storage system. The office data resources include unstructured log data, semi-structured business data, and structured indicator data. The office data resources are layered and merged into a data lake. Unstructured log data, semi-structured business data, and structured indicator data are imported into the source layer, standard layer, and aggregation layer of the data lake, respectively, to form cloud-native data assets with unified timestamps and metadata tags. High-dimensional feature vectors are extracted from the cloud-native data assets. These high-dimensional feature vectors are composed of business data attributes in the data lake standard layer, text semantic features in the source layer, and statistical indicators in the aggregation layer, in order to construct a feature engineering set for model training.

3. The distributed collaborative office data processing method based on artificial intelligence according to claim 2, characterized in that, The office data resources are layered and merged within a data lake, with unstructured log data, semi-structured business data, and structured indicator data imported into the source layer, standard layer, and aggregation layer of the data lake, respectively. Perform a row-delimiter-based splitting operation on the unstructured log data, convert the split log entries into key-value pair format with source system tags, and store them in the source-attached layer of the data lake; The pattern matching algorithm is applied to the semi-structured business data to compare the field names in the semi-structured business data with the predefined data dictionary of the data lake standard layer. Missing fields are filled in and redundant fields are removed to form a structured data table that conforms to the data lake standard layer specification. Time window aggregation is performed on the structured indicator data to uniformly convert indicator data with different time granularities to a statistical caliber based on hours, and data quality check bits are added to generate a summary data table in the data lake aggregation layer. Extracting high-dimensional feature vectors from the cloud-native data assets includes: One-hot encoding is performed on the business data attributes in the standard layer of the data lake to convert categorical variables into numerical sparse vectors; Word embedding technology is applied to the text semantic features in the data lake source layer to convert the text data into a dense vector of fixed dimensions; Max-min normalization is performed on the statistical indicators in the data lake aggregation layer to map indicator data of different dimensions to the same value range. The one-hot encoding results, word embedding results, and normalization results are concatenated along the feature dimension to form the high-dimensional feature vector.

4. The distributed collaborative office data processing method based on artificial intelligence according to claim 3, characterized in that, The feature-engineered set is input into a pre-built federated learning model for incremental training, including: The federated learning model maintains only global model parameters in the cloud and local training data on edge computing nodes. The network structure and hyperparameters of the federated learning model are initialized in the cloud, and the network structure and hyperparameters are distributed to each edge computing node participating in the training. Each edge computing node uses local data lake standard layer data to perform several rounds of forward and backward propagation on the federated learning model locally in order to calculate the update amount of local model parameters. Each edge computing node encrypts the update amount of its local model parameters and uploads it to the cloud. The cloud performs a weighted average of the update amounts from all edge computing nodes to update the global model parameters. The updated global model parameters are then distributed to each edge computing node to begin the next round of incremental training.

5. The distributed collaborative office data processing method based on artificial intelligence according to claim 4, characterized in that, The global inference model is used to identify abnormal behavior patterns in the cloud-native data assets, and the identified abnormal behavior patterns are correlated with the raw log data in the data lake's source layer, including: The high-dimensional feature vectors in cloud-native data assets are input into the global inference model, which outputs the probability value of each data sample belonging to normal or abnormal behavior. Samples with probability values ​​exceeding the anomaly threshold are filtered out, and their unique identifiers in the data lake are recorded. Based on the unique identifier, the corresponding original log data is retrieved from the data lake post source layer, and the operation user, operation object and operation time information in the original log data are extracted to generate the abnormal event snapshot.

6. The distributed collaborative office data processing method based on artificial intelligence according to claim 5, characterized in that, A causal graph is constructed on the snapshots of the abnormal events, and the relationship between the abnormal types and business entities in the standard layer of the data lake is mapped using knowledge graph technology, including: Using the exception type in the exception event snapshot as the head entity and the scope of impact involved in the exception event snapshot as the tail entity, establish a "cause" relationship edge in the graph database; Using business entities in the standard layer of the data lake as nodes and business dependencies between business entities as edges, construct a business entity relationship subgraph; The "cause" relationship edge is merged with the business entity relationship subgraph, and a path search is performed in the graph database to find the shortest propagation path from the anomaly type to the specific business entity. The shortest propagation path is the root cause analysis chain.

7. The distributed collaborative office data processing method based on artificial intelligence according to claim 6, characterized in that, Based on the root cause analysis chain, a data governance workflow is automatically triggered in the cloud computing platform, including: Analyze the business entities involved in the root cause analysis chain to determine the list of statistical indicators that need to be recalculated; A data quality recalculation task is created on the cloud computing platform. The data quality recalculation task reads the raw data of relevant statistical indicators in the data lake aggregation layer and cleans and repairs the raw data according to data quality rules. The cleaned and repaired data is recalculated to generate recalculated statistical indicators, which then overwrite the old indicator data in the data lake aggregation layer.

8. The distributed collaborative office data processing method based on artificial intelligence according to claim 7, characterized in that, The step of performing secondary feature fusion between the recalculated statistical indicators and the business data in the data lake standard layer to update the feature engineering set includes: The recalculated statistical indicators are joined with the business data in the standard layer of the data lake in the horizontal dimension to form a wide table containing historical and new features. Perform Pearson correlation analysis on the new features in the wide table to remove redundant new features that are highly correlated with existing features; The new features, after being filtered, are concatenated with the original high-dimensional feature vectors to form an updated feature engineering set.

9. The distributed collaborative office data processing method based on artificial intelligence according to claim 8, characterized in that, Also includes: During training, the computing resource consumption of the federated learning model on edge computing nodes is collected through cloud monitoring services. When the computing resource consumption exceeds the preset elastic scaling threshold, the number of container instances on the cloud platform is dynamically adjusted to allocate corresponding computing resources. When the federated learning model converges over multiple consecutive training cycles, the final global inference model is published as an online prediction service in the cloud to complete the intelligent processing of big data. During training, the cloud monitoring service collects data on the computing resource consumption of the federated learning model on edge computing nodes. When the computing resource consumption exceeds a preset elastic scaling threshold, the number of container instances on the cloud platform is dynamically adjusted, including: The cloud monitoring service periodically collects CPU utilization, memory usage, and network bandwidth usage on each edge computing node through an agent program. When the CPU utilization or memory usage exceeds the elastic scaling threshold, a scaling request is sent to the scheduling center of the cloud computing platform. The scheduling center then starts a new container instance in the cloud and migrates some computing tasks to the new container instance. When both CPU utilization and memory utilization are below the scaling-down threshold, a scaling-down request is sent to the scheduling center, which then stops some idle container instances to release computing resources.

10. The distributed collaborative office data processing method based on artificial intelligence according to claim 9, characterized in that, When the federated learning model converges over multiple consecutive training cycles, the final global inference model is published as an online prediction service in the cloud, including: After the federated learning model is trained, the global inference model is serialized, and the model's structure and parameters are saved as a model file in a specific format. Upload the model files to the cloud-based model repository, and configure the version number and access interface for the model files in the model repository; A prediction service instance is deployed in a container orchestration platform in the cloud. The prediction service instance loads model files from the model repository and provides a prediction interface based on the Hypertext Transfer Protocol to complete the intelligent processing of big data.

Images