Distributed artificial intelligence recommendation model training and application system for multi-platform data collaboration

Abstract

The application discloses a kind of distributed artificial intelligence recommendation model training and application system for multi-platform data cooperation, including data access and preprocessing layer, for safely accessing source data from multiple platforms, completing feature alignment locally, and adding noise to the aligned features based on the Laplace mechanism through the differential privacy algorithm, adding differential privacy noise based on the Laplace mechanism, taking into account data utility and privacy protection, dividing the client cluster through clustering algorithm, generating personalized global model for each cluster, thereby effectively avoiding the adaptability problem caused by unified global model, improving the model effect of different platforms. Through the combination of multi-agent collaboration network and cross-platform training engine technology, the training task is decoupled and unified computation abstraction is converted into matrix operation, global scheduling and intermediate result caching are realized, training time is greatly reduced, and large-scale multi-platform data training is supported.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a distributed artificial intelligence recommendation model training and application system for multi-platform data collaboration. Background Technology

[0002] A search revealed a method and apparatus for training a recommendation model, disclosed in Chinese Patent Publication No. CN119939238A. This method, based on the interaction between users and the recommendation system, trains a satisfaction prediction model using inverse reinforcement learning. This allows the model to capture user satisfaction across various interaction states. The method first predicts user satisfaction for each training data point, enabling the model to learn and recommend content with high user satisfaction during training, thereby improving the accuracy of the trained recommendation model.

[0003] However, the training method of this recommendation model only relies on user interaction data from a single platform to train the model. It cannot utilize heterogeneous data from multiple platforms or quickly adapt to multiple business scenarios. Furthermore, it only trains the satisfaction prediction model and recommendation model based on the full data of a single platform, without adopting a distributed computing framework or considering the inference needs under different scenarios. This easily leads to problems such as low model training efficiency and limited inference flexibility.

[0004] To address the shortcomings of existing technologies, it is necessary to design a distributed artificial intelligence recommendation model training and application system for multi-platform data collaboration. Summary of the Invention

[0005] The purpose of this invention is to address the shortcomings of existing technologies by proposing a distributed artificial intelligence recommendation model training and application system for multi-platform data collaboration.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: a distributed artificial intelligence recommendation model training and application system for multi-platform data collaboration, including a data access and preprocessing layer, for securely accessing source data from multiple platforms, completing feature alignment locally, and adding noise to the aligned features based on the Laplace mechanism using a differential privacy algorithm. The source data includes user data, item data, and context data.

[0007] The distributed model training layer employs heterogeneous federated learning and multi-agent collaboration. Models are trained locally on each client based on noisy feature data. The trained model parameters are encrypted and uploaded to the coordination server for personalized global model aggregation. This layer is based on a heterogeneous federated learning architecture with a clustering-based personalized model aggregation algorithm. An agent collaboration network and a cross-platform training engine are deployed on the heterogeneous federated learning architecture.

[0008] The collaborative reasoning and service layer dynamically merges local and global models, incorporates user feedback in real time, uses a generative architecture to create a recommendation list, and updates the local and global models. The recommendation list includes user information and recommended item information.

[0009] The application and interface layer is used for managing permissions and recommendation models, configuring connection parameters, receiving recommendation requests, and returning recommendation results.

[0010] Furthermore, the data access and preprocessing layer includes a multi-source data connector and a federated feature encoding and alignment module;

[0011] Multi-source data connectors provide adapter interfaces for source data from multiple platforms;

[0012] The federated feature encoding and alignment module encodes the source data locally based on the federated SVD algorithm and maps it to a unified vector space to achieve feature alignment. Through federated feature encoding technology, it enables secure data fusion across multiple platforms.

[0013] Furthermore, the specific method for implementing feature alignment in the federated feature encoding and alignment module is as follows:

[0014] Each platform locally calculates the approximate eigenvectors of the centralized matrix and covariance matrix, and encrypts the locally obtained local projection matrix using homomorphic encryption before sending it to the coordination server for computation. The coordination server aggregates all encrypted local projection matrices and constructs a global consensus projection space by calculating the principal directions of the local basis vectors. Based on the constructed global consensus projection space, the coordination server generates a global projection matrix through eigenvalue decomposition and distributes it to each platform. Each platform projects its original data features onto the global consensus projection space to obtain aligned features. The original data features include user data preprocessing features, item data preprocessing features, and context data preprocessing features.

[0015] Furthermore, the aggregation of the personalized global model specifically involves:

[0016] Each client trains a model using locally noise-added feature data. After adding differential privacy noise to the model parameters, the model is uploaded to the coordination server. The coordination server receives the trained model parameters from each client, forms a set, calculates the cosine similarity between the model parameters and constructs a similarity matrix. It then uses a clustering algorithm to identify clients with similar data distributions and divides the clients into several clusters. The coordination server aggregates a personalized global model for each cluster and distributes it to each client in the corresponding cluster. The client uses the distributed personalized global model as the initial value for the next round of training. The coordination server uses a weighted average to aggregate the personalized global model.

[0017] Furthermore, the agent collaboration network includes user-local representation agents, item-local representation agents, and global collaborative reasoning agents. User-local representation agents are deployed on the user platform, item-local representation agents are deployed on the item platform, and global collaborative reasoning agents are deployed on the coordination server. Recommendation requests are transformed into personalized global model training tasks and global collaborative reasoning tasks. A cross-platform training engine decomposes these tasks and assigns them to the corresponding agents, generating user-local feature vectors and item-local feature vectors. These feature vectors are then encrypted using an encrypted collaborative reasoning algorithm and uploaded to the global collaborative reasoning agent on the coordination server. The global collaborative reasoning agent calculates the feature vectors in an encrypted environment to obtain prediction results and loss gradients. These prediction results and loss gradients are then encrypted again and sent to the local agents. The user-local representation agents and item-local representation agents receive and decrypt the prediction results and loss gradients, and then update their local models accordingly. The prediction results include both user global representations and item global representations.

[0018] Furthermore, the cross-platform training engine is used to decompose personalized global model training tasks and global collaborative inference tasks into several sub-tasks and construct a computation graph; the optimizer calls the computation graph optimization algorithm to analyze the computation graph, perform logical optimization and physical optimization, uses a distributed task scheduling algorithm to call the computation framework, and uses an intelligent cache replacement algorithm to intelligently cache intermediate results, and unifies the training tasks into matrix operations to achieve global scheduling and optimization.

[0019] Furthermore, the collaborative reasoning and service layer includes a hybrid reasoning coordinator, a dynamic recommendation generator, and a feedback learning loop. The weights of the local and global models are dynamically adjusted based on the hybrid reasoning coordinator to ensure reasoning efficiency and recommendation accuracy.

[0020] The hybrid inference coordinator calculates weighted prediction scores based on recommendation requests using a dynamic weighted ensemble algorithm, and coordinates joint inference between the local model and the global collaborative model.

[0021] The dynamic recommendation generator generates a recommendation list based on user local representation, user global representation, item local representation, item global representation, contextual data, and weighted prediction scores. It adopts an Encoder-Decoder generative architecture to model the recommendation list problem as a sequence generation task, and then uses the Beam Search algorithm to find the sequence group with the highest probability to create the recommendation list.

[0022] The feedback learning loop acquires user behavior flow data and sends it to a message queue for temporary storage in real time. It uses online gradient descent to update the local model, directly calculating gradients and updating local model parameters using the latest user behavior data in the message queue as samples. For the global model, the coordination server collects interaction data in real time, calculates the model update amount using an incremental update algorithm, and sets the learning rate to incorporate it into the model. User features and item features are used as context, and the Bandit algorithm is used to assign each candidate recommended item as an Arm based on the context. The recommendation priority of each candidate item is dynamically adjusted based on user feedback to the recommendation results.

[0023] Furthermore, the application and interface layer includes a unified API gateway, an interpretation and recommendation module, a user permission management module, a model management module, and a system configuration module;

[0024] The unified API gateway receives external recommendation requests and uses an API key verification and matching algorithm to match the API key carried in the request with the legitimate keys stored in the system. It also checks the access permissions corresponding to the key. If the verification is successful, a token bucket algorithm is triggered for rate limiting. A data format standardization validation algorithm is used; if validation fails, an error message is returned. If validation is successful, the input data is encapsulated and sent to the collaborative reasoning and service layer to execute the recommendation logic. After receiving the returned results, the output data is processed again using a data format standardization algorithm to ensure a consistent format before being fed back to external applications.

[0025] The recommendation module explains how to populate user and item information from the recommendation list into a preset natural language template to generate intuitive and easy-to-understand recommendation explanations.

[0026] The user permission management module pre-stores the association rules between roles and operable permissions. Through a role-permission mapping matching algorithm, it assigns a corresponding role to each user and associates it with their executable operation permissions. When data is accessed from the data access and preprocessing layer, it configures data source connection parameters. The parameter validation algorithm verifies the validity of the parameter format in real time. Based on the configured data source parameters, it triggers a permission verification algorithm to check whether the current user role has the permission to load the corresponding adapter. Before the adapter reads the data metadata, the permission verification algorithm verifies again whether the user has the permission to read the data metadata. After successful verification, the adapter is allowed to obtain the data.

[0027] The model management module receives the trained recommendation model files, builds an index using a hash indexing algorithm, and records the recommendation model structure and training parameters. It calculates the hash value of the model file using the MD5 hash verification algorithm, compares it with the stored hash value of the original model, and after confirming that the model has not been tampered with, generates a release version and records a snapshot. It uses a version control algorithm to retrieve historical snapshots, quickly locates the corresponding model file based on the target version number, overwrites the currently active model, and updates the index information to achieve model rollback. Finally, it uses a B+ tree indexing algorithm to quickly retrieve the recommendation model structure, training parameters, and release version.

[0028] The system configuration module uses a sliding window statistical algorithm to collect system operation indicators in real time, analyze the current system load status and parameter adaptability, and calculates the optimal parameter combination based on the Bayesian optimization algorithm, with system performance indicators as the target and historical parameter adjustment data. It then uses a gradient descent tuning algorithm to fine-tune the parameter update step size and dynamically adjust the system parameters to achieve update optimization.

[0029] Furthermore, the multi-source data connector includes an adapter pool, a metadata management module, and a data monitoring module;

[0030] The adapter pool provides adapter interfaces for different data source types, and uses adapters to read data metadata, which includes data source connection information and schema structure.

[0031] The metadata management module is used to receive and store the connection information and schema structure of each data source in the data metadata, perform metadata lookup and matching, and record the data update frequency.

[0032] The data monitoring module reads connection information from various data sources and continuously monitors the connection status with different data sources.

[0033] Beneficial effects:

[0034] This invention adds differential privacy noise based on the Laplace mechanism, balancing data utility and privacy protection. It divides client clusters using a clustering algorithm, generating a personalized global model for each cluster, effectively avoiding compatibility issues caused by a unified global model and improving model performance across different platforms. By combining a multi-agent collaborative network with a cross-platform training engine, training tasks are decoupled and computational abstraction is uniformly transformed into matrix operations, enabling global scheduling and intermediate result caching, significantly reducing training time and supporting large-scale multi-platform data training. A four-layer architecture—data access, distributed training, collaborative inference, and application interface—decouples functional modules. When adding a recommendation task, only the recommendation generation logic needs adjustment, without reconstructing the overall architecture. It can be quickly integrated into different business systems through a unified API gateway, effectively improving scenario adaptability. Attached Figure Description

[0035] Figure 1 This is a schematic diagram of the system structure of the present invention;

[0036] Figure 2 This is a schematic diagram of the data access and preprocessing layer architecture of the present invention;

[0037] Figure 3 This is a schematic diagram of the distributed model training layer architecture of the present invention;

[0038] Figure 4 This is a schematic diagram of the collaborative reasoning and service layer architecture of the present invention. Detailed Implementation

[0039] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0040] Reference Figure 1-4 To achieve the above objectives, the present invention adopts the following technical solution: a distributed artificial intelligence recommendation model training and application system for multi-platform data collaboration, comprising:

[0041] The data access and preprocessing layer is used to securely access source data from multiple platforms, perform feature alignment locally, and add noise to the aligned features using a differential privacy algorithm based on the Laplace mechanism to satisfy differential privacy. The source data obtained in this embodiment includes:

[0042] User data includes: basic user attributes (such as age, gender, region, etc.) on each platform, user behavior data (such as interactive behavior flow data such as clicks, browsing, favorites, purchases, comments, and sharing), and user preference characteristics (such as historical areas of interest, consumption habits, and content interest tags, etc.).

[0043] Item data: basic attributes of items on each platform (such as product specifications, content categories, video duration, article themes, etc.), item association data (such as product category levels, content tag systems, item similarity associations, etc.), and item interaction data (such as item clicks, conversion rates, ratings, etc.).

[0044] Contextual data: the context of the recommendation request (such as access time, device type, network environment, etc.) and the associated features required for cross-platform collaboration (such as user cross-platform unique identifier mapping, item cross-platform category mapping, etc.).

[0045] The data access and preprocessing layer includes a multi-source data connector and a federated feature encoding and alignment module; the multi-source data connector includes an adapter pool, a metadata management module, and a data monitoring module.

[0046] The adapter pool provides adapter interfaces for different data source types, and uses adapters to read data metadata, which includes data source connection information and schema structure.

[0047] In this embodiment, the adapter pool adopts a parameter matching and type mapping algorithm. Based on the adapter pattern designed in software, it realizes data source access through the corresponding parameter matching and type mapping logic.

[0048] Specifically:

[0049] 1) Parameter configuration input: Configure connection parameters for the data source to be connected (including data source type, connection address, authentication information, etc.);

[0050] 2) Matching Mapping: The adapter pool has a built-in preset mapping relationship of "data source type - adapter". Through parameter matching and type mapping algorithm, the specific adapter that is compatible with the data source is selected from the adapter pool according to the configured data source type, connection protocol and other parameters.

[0051] 3) Connection and data reading: After successful matching, the corresponding adapter is loaded, a connection is established with the data source through the adapter interface and permissions are verified, and finally the metadata of the data source is read through the adapter to realize unified interface access for multi-source data.

[0052] The metadata management module is used to receive and store the connection information and schema structure of each data source in the data metadata, perform metadata lookup and matching, and record the data update frequency.

[0053] In this embodiment, the metadata management module employs two basic data management algorithms: a metadata indexing algorithm and a data update frequency statistics algorithm. The metadata indexing algorithm includes commonly used B+ tree indexes and hash indexes for fast metadata retrieval. The data update frequency statistics algorithm is a sliding window-based statistical algorithm used to calculate the data source update frequency in real time. Specifically:

[0054] Metadata storage and index building: After receiving the data source connection information and schema structure uploaded by the adapter, the metadata management module uses the B+ tree index algorithm to build indexes on core fields (including data source ID and table name) to achieve fast metadata lookup and matching.

[0055] Update frequency statistics: Using a sliding window algorithm, a fixed time window is set, and the number of times the data source is updated within the window is counted, automatically calculating and updating the data update frequency index.

[0056] Metadata Query and Maintenance: When other modules of the system request metadata, the target metadata is quickly located through hash index or B+ tree index and the query results are returned; at the same time, invalid metadata is checked regularly through indexing algorithms to ensure data accuracy.

[0057] The data monitoring module reads connection information from various data sources and continuously monitors the connection status with different data sources.

[0058] In this embodiment, the data monitoring module employs heartbeat detection, timeout reconnection, and anomaly threshold judgment algorithms from the fields of network communication and data integration. The heartbeat detection algorithm verifies the connection validity by periodically sending detection packets and is a classic technique for monitoring network connection status. The timeout reconnection algorithm automatically triggers reconnection logic in connection interruption scenarios and is applied to multi-source data access systems. The anomaly threshold judgment algorithm identifies anomalies based on preset thresholds.

[0059] Specifically, when the data monitoring module starts, it reads the connection information (IP, port, communication protocol) of each data source, sets the heartbeat detection period, timeout threshold, and reconnection limit, and sends lightweight detection packets to each data source according to the set period through the heartbeat detection algorithm, waiting for the data source to respond. If a response is received within the timeout threshold, the connection is determined to be normal, and the current status is recorded; if no response is received, the timeout reconnection algorithm is triggered, and the connection is automatically re-established until the reconnection limit is reached. When the reconnection fails or the number of connection failures reaches the preset abnormal threshold, the abnormal threshold judgment algorithm marks the data source as "connection abnormal", the abnormal log is recorded synchronously, and the system alarm mechanism is triggered (such as notifying the administrator).

[0060] The federated feature encoding and alignment module encodes the source data locally based on the federated SVD algorithm and maps it to a unified vector space to achieve feature alignment.

[0061] Specifically:

[0062] Each platform locally calculates the approximate eigenvectors of the centralized matrix and covariance matrix, and encrypts the locally obtained local projection matrix using homomorphic encryption before sending it to the coordination server for computation. The coordination server aggregates all encrypted local projection matrices and constructs a global consensus projection space by calculating the principal directions of the local basis vectors. Based on the constructed global consensus projection space, the coordination server generates a global projection matrix through eigenvalue decomposition and distributes it to each platform. Each platform projects its original data features onto the global consensus projection space to obtain aligned features. The original data features include user data preprocessing features, item data preprocessing features, and context data preprocessing features, which correspond to the source data of multiple platforms. These are the original features of the source data that have been preprocessed (centralized) locally on each platform but have not been aligned across platforms. The core feature is the set of original feature dimensions of the platform's own data.

[0063] Features after alignment Alternatively, Laplacian noise can be added to the gradient parameters used for training to satisfy ϵ-differential privacy, as follows:

[0064]

[0065] in It is noise sampled from a Laplace distribution, with a scale parameter of . z is the value that needs to be protected (including eigenvalues ​​or model gradients). Indicates the sensitivity of the function; When used for eigenvalues, it is usually the maximum range of variation of that eigenvalue; When used for gradients, it is usually the upper limit threshold of the gradient norm; To budget for privacy, control the intensity of privacy protection. The smaller the value, the greater the added noise, the stronger the privacy protection, and the lower the data utility;

[0066] In features Before leaving the local platform, sensitive fields involving individual user information are classified according to their sensitivity. and the set privacy budget Add Laplace noise.

[0067] In this embodiment, the federated feature encoding and alignment module uses the federated SVD algorithm for feature alignment. The specific steps are as follows:

[0068] for There are 1 platform, and each platform has a local feature matrix. (User-feature matrix), but with different feature dimensions. The goal of the federated SVD algorithm is to find a shared latent space such that all... All can be projected into this space:

[0069] 1) Local SVD calculation: for each platform Calculate the matrices after data centralization of their respective source data locally. covariance matrix The approximate Top-L feature vector is expressed as:

[0070]

[0071] in Indicates platform Locally centralized data matrix ( One user, (characteristics) Indicates inclusion The first L principal eigenvector matrices, where L is the dimension of the unified latent space defined by the coordinating server;

[0072] 2) Secure aggregation: Each platform will aggregate the feature vector matrix obtained locally. The encrypted text is sent to the coordinating server using homomorphic encryption, and the server is unable to decrypt it to obtain the plaintext. , and perform specific calculations on the ciphertext;

[0073] 3) Global consensus space construction: The coordination server aggregates all encrypted data. The matrix is ​​used to construct a global consensus projection space V by calculating the principal directions of the local basis vectors;

[0074] One common approach is to construct an aggregation matrix A, denoted as... Then, perform eigenvalue decomposition on A, and take the first L eigenvectors to form the global projection matrix. ,in ;

[0075] Another approach is to directly approximate the global principal components using an iterative algorithm in the encrypted state;

[0076] 4) Local Feature Projection: The coordination server distributes the global projection matrix V to each platform, and each platform k projects its own original features. Projecting onto this global space yields the aligned feature representation:

[0077]

[0078] in It is the original feature dimension of the global matrix V relative to the platform k. The corresponding blocks, It is a new feature matrix with unified dimensions after platform k-alignment.

[0079] The distributed model training layer employs heterogeneous federated learning and multi-agent collaboration. Models are trained locally on each client using noisy feature data. The trained model parameters are then encrypted and uploaded to a coordination server for aggregation of personalized global models. This layer is based on a heterogeneous federated learning architecture, which deploys an agent collaboration network and a cross-platform training engine. The heterogeneous federated learning architecture uses a clustering-based personalized model aggregation algorithm to identify clients with similar data distributions and generate a personalized global model for each cluster.

[0080] Specifically:

[0081] Each client trains a model using locally noise-added feature data, encrypts the model parameters, and uploads them to the coordination server. The coordination server receives the trained model parameters from each client, forms a set, calculates the cosine similarity between the model parameters, constructs a similarity matrix, and uses a clustering algorithm to identify clients with similar data distributions, dividing the clients into several clusters. The coordination server aggregates a personalized global model for each cluster and distributes it to each client in the corresponding cluster. The client uses the distributed personalized global model as the initial value for the next round of training. The coordination server uses a weighted average to aggregate the personalized global model.

[0082] In this embodiment, the heterogeneous federated learning module uses a cluster-based personalized model aggregation algorithm to generate a global model. The specific steps are as follows:

[0083] 1) Each client p uses local data Training your own model The goal is to minimize the local loss function. , represented as:

[0084]

[0085] Where t is the communication round; The learning rate; This represents the gradient of the local loss function;

[0086] 2) Model Upload: The client uploads the trained model parameters. Uploaded to the coordination server, parameters can be encrypted or differential privacy noise can be added to protect privacy.

[0087] 3) Server-side clustering: The server receives the model parameter sets from all clients, calculates the cosine similarity between each pair of model parameters, constructs a similarity matrix, and then the clustering algorithm divides the clients into C clusters, where clients with high similarity are considered to have similar data distributions.

[0088] 4) Instead of performing a global average, the server aggregates a personalized global model for each cluster separately. This aggregation typically uses a weighted average, expressed as:

[0089]

[0090] in Let k be the number of data samples for client k. For the c-th cluster Total sample size of all clients and ;

[0091] 5) Model distribution: The server distributes the corresponding personalized global model to each client in the cluster;

[0092] 6) Local adaptation: After receiving their respective personalized global models, the clients use them as the initial values ​​for the next round of training, transferring the knowledge of the global model to their own local models.

[0093] The intelligent agent collaborative network comprises user-local representation agents, item-local representation agents, and global collaborative reasoning agents. User-local representation agents are deployed on the user platform, item-local representation agents on the item platform, and global collaborative reasoning agents on a coordination server. Several agents collaborate to transform recommendation requests into personalized global model training tasks and global collaborative reasoning tasks. A cross-platform training engine decomposes and assigns these tasks to the corresponding agents, decoupling user representation learning from item representation learning. This generates user-local feature vectors and item-local feature vectors. These feature vectors are then encrypted using an encrypted collaborative reasoning algorithm and uploaded to the global collaborative reasoning agent on the coordination server. The global collaborative reasoning agent calculates the feature vectors in an encrypted environment, obtaining prediction results and loss gradients. These results are then encrypted again and sent back to the local agents. The user-local representation agents and item-local representation agents receive and decrypt the prediction results and loss gradients, updating their local models accordingly. The prediction results include both user global representations and item global representations.

[0094] In this embodiment, the agent collaboration network and the cross-platform training engine collaborate to achieve the training task of a personalized global model; specifically:

[0095] 1) Task initialization and decomposition: The personalized global model training task and the global collaborative reasoning task are decomposed into three sub-tasks through the cross-platform training engine: user local representation, item local representation, and global collaborative reasoning, which are then assigned to the corresponding agents.

[0096] 2) Local representation learning: The user local representation agent on the user platform trains the model based on local user data through a distributed representation learning algorithm to generate user feature vectors; the item local representation agent on the item platform generates item feature vectors in the same way.

[0097] 3) Encrypted secure upload: The two local agents encrypt the generated feature vectors using an encrypted collaborative reasoning algorithm and then upload them to the global collaborative reasoning agent in the coordination server to avoid privacy leaks caused by plaintext transmission.

[0098] 4) Collaborative Reasoning and Feedback: The global collaborative reasoning agent calculates two feature vectors in an encrypted environment to obtain the prediction result and loss gradient. Then, the prediction result and loss gradient are sent to the corresponding local agent through an encrypted algorithm. The prediction result includes the global representation of the user and the global representation of the item.

[0099] 5) Local Model Update: After receiving and decrypting the prediction results and loss gradients, the user's local representation agent and the item's local representation agent update their respective local models, completing one round of collaborative learning and iteratively optimizing the representation accuracy.

[0100] A cross-platform training engine is used to decompose personalized global model training tasks and global collaborative inference tasks into several sub-tasks and construct a computation graph. The optimizer calls the computation graph optimization algorithm to analyze the computation graph, perform logical optimization and physical optimization, uses a distributed task scheduling algorithm to call the computation framework, and uses an intelligent cache replacement algorithm to intelligently cache intermediate results. The training tasks are uniformly abstracted into matrix operations to achieve global scheduling and optimization.

[0101] Specifically:

[0102] 1) Task decomposition and computation graph construction: After receiving the personalized global model training task and the global collaborative reasoning task, the complex training task is broken down into several sub-tasks (such as user representation calculation and matrix multiplication sub-task) through the basic task decomposition algorithm, and the corresponding computation graph is constructed to clearly present the dependencies between sub-tasks.

[0103] 2) Computation graph optimization: The optimizer calls the computation graph optimization algorithm to first perform logical optimization, eliminating redundant nodes in the computation graph and merging related operators; then it performs physical optimization, planning the optimal execution order according to the task type (such as matrix operations) to improve computational efficiency.

[0104] 3) Distributed task scheduling: Through a distributed task scheduling algorithm, the computational requirements (such as computing power and memory) of each subtask are analyzed, and the underlying system automatically calls the appropriate computing framework (GPU framework suitable for matrix operations, CPU framework suitable for lightweight computing) while balancing the resource load of each node.

[0105] 4) Intelligent caching of intermediate results: An intelligent caching replacement algorithm is used to cache high-frequency intermediate results during the calculation process; when subsequent subtasks need to call the result, it is read directly from the cache, avoiding repeated calculations and greatly reducing training time.

[0106] The collaborative reasoning and service layer includes a hybrid reasoning coordinator, a dynamic recommendation generator, and a feedback learning loop. It dynamically merges local and global models, incorporates user feedback in real time, creates recommendation lists using a generative architecture, and updates local and global models.

[0107] The hybrid inference coordinator calculates weighted prediction scores based on recommendation requests using a dynamic weighted ensemble algorithm, and coordinates joint inference between the local model and the global collaborative model.

[0108] The hybrid inference coordinator utilizes both local and global collaborative models for candidate items. Its final prediction score The calculation formula is:

[0109]

[0110] in Let j be the predicted score of item j on the local model. Let j be the predicted score of item j on the global collaborative model. These are dynamic weighting coefficients, ranging from [0, 1] and... c represents the context feature vector of the current request, w is the weight vector, and b is the bias term. The Sigmoid function compresses the output to the (0,1) interval.

[0111] When users are very active and in sessions with intensive real-time feedback, λ approaches 1, and the system trusts the local model more to pursue low latency and immediacy; when users have not logged in for several days or request cross-platform content, λ approaches 0, and the system relies more on the global model.

[0112] The dynamic recommendation generator generates a recommendation list based on user local representation, user global representation, item local representation, item global representation, contextual data, and weighted prediction scores. It transforms the traditional requirement of recommending suitable items to users into a ranking based on user interest probabilities. It adopts an Encoder-Decoder generative architecture to model the recommendation list problem as a sequence generation task, and then uses the Beam Search algorithm to find the sequence group with the highest probability to create the recommendation list. The recommendation list includes user information and recommended item information.

[0113] The reasoning process of the Encoder-Decoder generative architecture of the dynamic recommendation generator includes:

[0114] 1) The Encoder encodes the user's intent, and its formula is expressed as:

[0115]

[0116]

[0117] in It is a local representation for the user, derived from the user's intelligent agent; The user's global representation comes from the collaborative intelligent agent; This represents the context feature vector of the current request. H is the user's historical behavior sequence (i.e., item ID sequence), and H is the comprehensive user state representation output by the Encoder, used to encode the user's current intent.

[0118] 2) The Decoder generates an ordered sequence of recommended item IDs through autoregression. In each step t, set:

[0119]

[0120]

[0121] in The item ID generated in the previous step. Let this be the hidden state of the Decoder's output in step t. , This represents the parameter matrix that maps the hidden states to the probability distribution of the entire item library. This indicates that, given the user's history and intent, the next step is to generate an item. The probability of.

[0122] The feedback learning loop acquires user behavior flow data and sends it to a message queue for temporary storage in real time, ensuring the stability and real-time nature of data transmission. The online gradient descent method is used to update the local model. Using the latest user behavior data in the message queue as a sample, the gradient is directly calculated and the local model parameters are updated to achieve real-time iteration of the local model and quickly adapt to the user's recent preferences.

[0123] For the global model, the coordination server collects interaction data in real time, calculates the model update amount through an incremental update algorithm, and sets a learning rate to incorporate into the model. This embodiment uses a small step learning rate to achieve incremental updates, avoiding the high time consumption and resource consumption caused by full retraining. User features (historical preferences, current state) and item features (category, attributes) are used as context. Based on the context, each candidate recommended item is used as an Arm through the Bandit algorithm. The recommendation priority of each candidate item is dynamically adjusted according to the user's response to the recommendation results (clicks are positive feedback, ignores are negative feedback), the recommendation strategy is optimized, and the model is adjusted according to the latest user behavior to achieve online learning and evolution. Behavioral flow data includes clicks, ignores, favorites, etc.

[0124] Among them, the Online Gradient Descent (OGD) algorithm is a real-time model update algorithm for streaming data that does not require batch data and can quickly respond to new samples; the incremental update algorithm incorporates the amount of new data updates by using a small step learning rate, avoiding full model retraining; and the Bandit algorithm combines user and item features (context) to optimize the recommendation strategy.

[0125] The application and interface layer includes a unified API gateway, an interpretation and recommendation module, a user permission management module, a model management module, and a system configuration module. It is used for permission and recommendation model management, connection parameter configuration, receiving recommendation requests, and returning recommendation results. Recommendation requests are triggered by, for example, the call requirements of the "Homepage Product Recommendation" and "You May Like" modules of an e-commerce platform, or by the related recommendations triggered after the user refreshes the content homepage, clicks "Change Batch" recommendation list, or completes a purchase / browse.

[0126] The unified API gateway receives external recommendation requests and uses an API key verification and matching algorithm to match the API key carried in the request with the legitimate keys stored in the system. It also checks the access permissions corresponding to the key (e.g., whether calling the recommendation interface is allowed). If there is a mismatch or insufficient permission, the request is rejected directly. If the verification is successful, a token bucket algorithm is triggered for rate limiting. A data format standardization validation algorithm is used; if validation fails, an error message is returned. If validation is successful, the input data is encapsulated and forwarded to the collaborative reasoning and service layer to execute the recommendation logic. After receiving the returned results, the output data is processed again using a data format standardization algorithm to ensure a consistent format before being fed back to the external application.

[0127] The recommendation module fills in the user information and recommended item information in the recommendation list into a preset natural language template to generate an intuitive and easy-to-understand recommendation explanation, such as "We recommend XX for you because you have previously followed XX, XX and other related items";

[0128] The user permission management module pre-stores the association rules between roles and operable permissions. Through a role-permission mapping matching algorithm, it assigns a corresponding role to each user and associates it with their executable operation permissions (data source configuration permissions, adapter access permissions). When data is accessed from the data access and preprocessing layer data access, data source connection parameters (address, authentication information) are configured. A parameter validation algorithm verifies the validity of the parameter format (IP format, port range) in real time to ensure the configuration parameters are valid. Based on the configured data source parameters, a permission verification algorithm is triggered to check whether the current user role has the permission to load the corresponding adapter. If no permission is granted, loading is refused; otherwise, the system is allowed to call the adapter interface. Before the adapter reads data metadata, the permission verification algorithm again verifies whether the user has the permission to read the data metadata. After successful verification, the adapter is allowed to obtain the data metadata, and a permission operation log is recorded to ensure full traceability. This role-based access control model implements permission operation control.

[0129] The model management module receives the trained recommendation model files and builds an index using a hash indexing algorithm. In this embodiment, the index is built based on core fields such as model ID and version number. It also records the recommendation model structure and training parameters, supporting fast lookups. When a model is released, the hash value of the model file is calculated using the MD5 hash verification algorithm and compared with the stored hash value of the original model. After confirming that the model has not been tampered with, a release version is generated and a snapshot is recorded. When a model needs to be rolled back, a version control algorithm is used to retrieve historical snapshots, quickly locate the corresponding model file based on the target version number, overwrite the currently active model, and update the index information to achieve model rollback, ensuring that the system uses the rolled-back model. The module also uses a B+ tree indexing algorithm to quickly retrieve the recommendation model structure, training parameters, and release version, intuitively displaying the model's history and current status.

[0130] The system configuration module uses a sliding window statistical algorithm to collect system operation metrics (CPU / GPU utilization, response latency, QPS) in real time, analyzes the current system load status and parameter suitability, and calculates the optimal parameter combination (training task parallelism, cache size, request rate limiting threshold) based on the Bayesian optimization algorithm, with the goal of reducing system performance metrics (reducing latency and increasing throughput). It also uses historical parameter adjustment data to fine-tune the parameter update step size through the gradient descent tuning algorithm to avoid system instability caused by parameter mutations, and dynamically adjusts system parameters to achieve update optimization.

[0131] The AI ​​recommendation model training and application system is based on a component architecture platform. By coordinating the data value of multiple data platforms (e-commerce, social, and content platforms), it trains a more powerful and accurate AI recommendation model without requiring the data to leave its home environment or domain. Ultimately, the model is applied to provide users with personalized services across platforms. It can find a balance between enjoying the intelligent benefits of data and protecting personal privacy rights. It is one of the key infrastructures for promoting the continuous and healthy development of the digital economy and has extremely strong application prospects.

[0132] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A distributed artificial intelligence recommendation model training and application system for multi-platform data collaboration, characterized in that, include: The data access and preprocessing layer is used to securely access source data from multiple platforms, perform feature alignment locally, and add noise to the aligned features based on the Laplace mechanism using a differential privacy algorithm. The source data includes user data, item data, and context data. The distributed model training layer employs heterogeneous federated learning and multi-agent collaboration. Models are trained locally on each client based on noisy feature data. The trained model parameters are encrypted and uploaded to the coordination server for personalized global model aggregation. This layer is based on a heterogeneous federated learning architecture with a clustering-based personalized model aggregation algorithm. An agent collaboration network and a cross-platform training engine are deployed on the heterogeneous federated learning architecture. The collaborative reasoning and service layer dynamically merges local and global models, incorporates user feedback in real time, uses a generative architecture to create a recommendation list, and updates the local and global models. The recommendation list includes user information and recommended item information. The application and interface layer is used for managing permissions and recommendation models, configuring connection parameters, receiving recommendation requests, and returning recommendation results.

2. The distributed artificial intelligence recommendation model training and application system according to claim 1, characterized in that: The data access and preprocessing layer includes a multi-source data connector and a federated feature encoding and alignment module; Multi-source data connectors provide adapter interfaces for source data from multiple platforms; The federated feature encoding and alignment module encodes the source data locally based on the federated SVD algorithm and maps it to a unified vector space to achieve feature alignment.

3. The distributed artificial intelligence recommendation model training and application system according to claim 1, characterized in that: The specific method for implementing feature alignment in the federated feature encoding and alignment module is as follows: Each platform locally calculates the approximate eigenvectors of the centralized matrix and covariance matrix, and encrypts the locally obtained local projection matrix using homomorphic encryption before sending it to the coordination server for computation. The coordination server aggregates all encrypted local projection matrices and constructs a global consensus projection space by calculating the principal directions of the local basis vectors. Based on the constructed global consensus projection space, the coordination server generates a global projection matrix through eigenvalue decomposition and distributes it to each platform. Each platform projects its original data features onto the global consensus projection space to obtain aligned features. The original data features include user data preprocessing features, item data preprocessing features, and context data preprocessing features.

4. The distributed artificial intelligence recommendation model training and application system according to claim 1, characterized in that: The aggregation of the personalized global model specifically involves: Each client trains a model using locally noise-added feature data. After adding differential privacy noise to the model parameters, the model is uploaded to the coordination server. The coordination server receives the trained model parameters from each client, forms a set, calculates the cosine similarity between the model parameters and constructs a similarity matrix. It then uses a clustering algorithm to identify clients with similar data distributions and divides the clients into several clusters. The coordination server aggregates a personalized global model for each cluster and distributes it to each client in the corresponding cluster. The client uses the distributed personalized global model as the initial value for the next round of training. The coordination server uses a weighted average aggregation of personalized global models.

5. The distributed artificial intelligence recommendation model training and application system according to claim 4, characterized in that: The agent collaboration network includes user-local representation agents, item-local representation agents, and global collaborative reasoning agents. User-local representation agents are deployed on the user platform, item-local representation agents are deployed on the item platform, and global collaborative reasoning agents are deployed on the coordination server. Recommendation requests are transformed into personalized global model training tasks and global collaborative reasoning tasks. A cross-platform training engine decomposes these tasks and assigns them to the corresponding agents, generating user-local feature vectors and item-local feature vectors. These feature vectors are then encrypted using an encrypted collaborative reasoning algorithm and uploaded to the global collaborative reasoning agent on the coordination server. The global collaborative reasoning agent calculates the feature vectors in an encrypted environment to obtain prediction results and loss gradients. These results are then encrypted again and sent back to the local agents. The user-local representation agents and item-local representation agents receive and decrypt the prediction results and loss gradients, and then update their local models accordingly. The prediction results include both user global representations and item global representations.

6. The distributed artificial intelligence recommendation model training and application system according to claim 5, characterized in that: The cross-platform training engine is used to decompose personalized global model training tasks and global collaborative inference tasks into several sub-tasks and construct a computation graph. The optimizer calls the computation graph optimization algorithm to analyze the computation graph, perform logical optimization and physical optimization, uses a distributed task scheduling algorithm to call the computation framework, and uses an intelligent cache replacement algorithm to intelligently cache intermediate results, and unifies the training tasks into matrix operations to achieve global scheduling and optimization.

7. The distributed artificial intelligence recommendation model training and application system according to claim 5, characterized in that: The collaborative reasoning and service layer includes a hybrid reasoning coordinator, a dynamic recommendation generator, and a feedback learning loop. The hybrid inference coordinator calculates weighted prediction scores based on recommendation requests using a dynamic weighted ensemble algorithm, and coordinates joint inference between the local model and the global collaborative model. The dynamic recommendation generator generates a recommendation list based on user local representation, user global representation, item local representation, item global representation, contextual data, and weighted prediction scores. It adopts an Encoder-Decoder generative architecture to model the recommendation list problem as a sequence generation task, and then uses the Beam Search algorithm to find the sequence group with the highest probability to create the recommendation list. The feedback learning loop acquires user behavior flow data and sends it to a message queue for temporary storage in real time. It uses online gradient descent to update the local model, directly calculating gradients and updating local model parameters using the latest user behavior data in the message queue as samples. For the global model, the coordination server collects interaction data in real time, calculates the model update amount using an incremental update algorithm, and sets the learning rate to incorporate it into the model. User features and item features are used as context, and the Bandit algorithm is used to assign each candidate recommended item as an Arm based on the context. The recommendation priority of each candidate item is dynamically adjusted based on user feedback to the recommendation results.

8. The distributed artificial intelligence recommendation model training and application system according to claim 1, characterized in that: The application and interface layer includes a unified API gateway, an interpretation and recommendation module, a user permission management module, a model management module, and a system configuration module; The unified API gateway receives external recommendation requests and uses an API key verification and matching algorithm to match the API key carried in the request with the legitimate keys stored in the system. It also checks the access permissions corresponding to the key. If the verification is successful, a token bucket algorithm is triggered for rate limiting. A data format standardization validation algorithm is used; if validation fails, an error message is returned. If validation is successful, the input data is encapsulated and sent to the collaborative reasoning and service layer to execute the recommendation logic. After receiving the returned results, the output data is processed again using a data format standardization algorithm to ensure a consistent format before being fed back to external applications. The recommendation module explains how to populate user and item information from the recommendation list into a preset natural language template to generate intuitive and easy-to-understand recommendation explanations. The user permission management module pre-stores the association rules between roles and operable permissions. Through the role-permission mapping matching algorithm, it assigns a corresponding role to each user and associates it with the operable permissions. When data is accessed and preprocessed, it configures the data source connection parameters. The parameter validation algorithm verifies the validity of the parameter format in real time. Based on the configured data source parameters, it triggers the permission verification algorithm to check whether the current user role has the permission to load the corresponding adapter. Before the adapter reads the data metadata, the permission verification algorithm verifies again whether the user has permission to read the data metadata. If the verification is successful, the adapter is allowed to obtain the data. The model management module receives the trained recommendation model files, builds an index using a hash indexing algorithm, and records the recommendation model structure and training parameters. It calculates the hash value of the model file using the MD5 hash verification algorithm, compares it with the stored hash value of the original model, and after confirming that the model has not been tampered with, generates a release version and records a snapshot. It uses a version control algorithm to retrieve historical snapshots, quickly locates the corresponding model file based on the target version number, overwrites the currently active model, and updates the index information to achieve model rollback. Finally, it uses a B+ tree indexing algorithm to quickly retrieve the recommendation model structure, training parameters, and release version.

9. The system configuration module uses a sliding window statistical algorithm to collect system operation indicators in real time, analyze the current system load status and parameter adaptability, and calculate the optimal parameter combination based on the Bayesian optimization algorithm, with system performance indicators as the target and combined with historical parameter adjustment data. The gradient descent tuning algorithm is used to fine-tune the parameter update step size, and the system parameters are dynamically adjusted to achieve update optimization.

10. The distributed artificial intelligence recommendation model training and application system according to claim 1, characterized in that: The multi-source data connector includes an adapter pool, a metadata management module, and a data monitoring module; The adapter pool provides adapter interfaces for different data source types, and uses adapters to read data metadata, which includes data source connection information and schema structure. The metadata management module is used to receive and store the connection information and schema structure of each data source in the data metadata, perform metadata lookup and matching, and record the data update frequency. The data monitoring module reads connection information from various data sources and continuously monitors the connection status with different data sources.

Images