Distributed user portrait construction method and system based on big data

Abstract

The invention discloses a distributed user portrait construction method and system based on big data, and relates to the technical field of big data, the system is composed of a plurality of functional modules, and the system comprises a multi-dimensional fragmentation storage module for obtaining multi-modal dimension data, constructing a dynamic weight and multi-constraint fragmentation system based on the multi-modal dimension data, and storing the dynamic weight and multi-constraint fragmentation system based on the multi-modal dimension data; an elastic adjustment mechanism is introduced to carry out fragmentation storage on the multi-modal dimension data to obtain distributed data; the privacy protection module is used for obtaining candidate feature data and global tag data based on the multi-modal dimension data, and obtaining a feature contribution degree by calculating the mutual information sum and proportion of the candidate feature data and the global tag data; carrying out local layered screening according to the distributed data to obtain discriminative features, and carrying out weighted fusion on the discriminative features and the feature contribution degrees to output a high-quality feature set; and the offline collaboration module is used for training an average model through an improved FedAvg algorithm based on the high-quality feature set.

Description

Technical Field

[0001] This invention relates to the field of big data technology, specifically to a distributed user profile construction method and system based on big data. Background Technology

[0002] User profiling is a structured set of user characteristics formed by integrating, analyzing, and tagging multi-dimensional information with data at its core. Essentially, it is a virtual and precise mapping of users. In the digital age, user behavior has largely migrated online, making it difficult for businesses to have face-to-face contact with users. By describing user profiles, user needs can be restored from multiple dimensions, bridging the information gap that cannot be directly communicated, guiding product design, clarifying the direction of new product development, optimizing user experience, integrating scattered user data to form a unified data view, providing a structured foundation for data analysis, and transforming vague user perceptions into quantifiable characteristics to avoid blind decision-making.

[0003] Existing technologies face multiple bottlenecks: First, the problem of data silos is prominent, with multi-source and multimodal data scattered across distributed nodes. Due to privacy compliance constraints, centralized processing is difficult, leading to insufficient data value mining. Second, under the federated learning framework, the traditional FedAvg algorithm adopts a simple averaging strategy, which is susceptible to malicious attacks from Byzantine nodes or interference from low-quality data, resulting in insufficient model robustness. Furthermore, it does not consider the differences in node feature contributions, affecting the accuracy of user profiles. Third, there is an inherent contradiction between the low latency requirements of real-time tasks and the high-precision computation requirements of offline tasks. Existing systems struggle to dynamically balance these two aspects, leading to an imbalance between the timeliness and accuracy of user profiles. Fourth, feature processing is inefficient. Insufficient standardization of multimodal data, ineffective filtering of redundant features, and a lack of dynamic weight adaptation mechanisms prevent the accurate capture of dynamic changes in user behavior, further reducing the quality of user profiles. Summary of the Invention

[0004] To achieve the above objectives, the present invention provides the following technical solution: A distributed user profiling system based on big data includes: The multi-dimensional sharding storage module acquires multi-modal dimensional data, constructs a dynamic weight and multi-constraint sharding system based on the multi-modal dimensional data, and introduces an elastic adjustment mechanism to shard and store the multi-modal dimensional data to obtain distributed data; The privacy protection module acquires candidate feature data and global label data based on multimodal dimensional data. It calculates the sum and proportion of mutual information between candidate feature data and global label data to obtain the feature contribution. It performs local hierarchical screening based on distributed data to obtain discriminative features. It then weights and fuses the discriminative features with the feature contribution to output a high-quality feature set. The offline collaboration module trains an average model based on a high-quality feature set by improving the FedAvg algorithm, and outputs a user profile based on the trained average model combined with dynamic weights. Among them, the improved FedAvg algorithm introduces node contribution for gradient weighting updates on the basis of the federated averaging framework, and filters low-quality node gradients through Byzantine behavior penalty, replacing the average strategy in the federated averaging framework.

[0005] Furthermore, the process of acquiring multimodal dimensional data is as follows: Through a distributed data acquisition system, five categories of data—identity, space, attribute, behavior, and time—are selectively captured from the entire user interaction chain and business systems. After standardization processing, these data are transformed into multimodal dimensional data in a unified format.

[0006] Furthermore, the process of constructing a dynamic weighted and multi-constraint partitioning system is as follows: Multimodal dimensional data is normalized and metadata is extracted. Based on the normalized multimodal dimensional data and metadata, the weights of the multimodal dimensional data at time t are calculated according to a preset dynamic weight function. The dynamic weights of each dimension are obtained by weighted fusion using the balance coefficients determined by the analytic hierarchy process. A sharding value function is constructed based on dynamic weights and metadata. The entropy weight method is used to determine the index weights, and the comprehensive value of the sharding scheme is obtained by weighting. Based on comprehensive value, a multi-group partitioning scheme is generated by using a preset combination of dimensional partitioning thresholds as individuals in the population. The inverse of the partitioning value function is used as the fitness function, and the optimal partitioning scheme is output through iterative optimization by roulette wheel selection, dimensional threshold crossover, weight or threshold mutation genetic operations.

[0007] Furthermore, the process of acquiring distributed data is as follows: Using standardized multimodal dimensional data as the processing object, an improved genetic algorithm is introduced based on dynamic weights to solve a multi-constraint optimal sharding scheme. First, the multi-constraint optimal sharding scheme is implemented to split the data and bind it to distributed nodes, initial storage shards are created and metadata is recorded. Then, the load of distributed nodes, query latency and dynamic indicators of initial storage shards are monitored. If the initial storage shards do not reach the preset threshold, the initial storage shards are dynamically adjusted. Finally, the dynamic indicators are verified to meet the standards, forming distributed data adapted to dynamic scenarios.

[0008] Furthermore, the process of obtaining candidate feature data and global label data is as follows: Multimodal dimension data under jurisdiction is extracted from distributed data, and standardized local features are obtained after adaptive standardization through a sliding window. Based on the pre-acquired local business labels, the mutual information between the standardized local features and the local business labels is calculated. A dynamic threshold is obtained based on the number of users, and candidate feature data is obtained by combining the mutual information and the dynamic threshold. Local business tags are encrypted and uploaded via Paillier using distributed data. The encrypted tags are then aggregated and decrypted using the additive homomorphic property of the distributed data to obtain global tag data.

[0009] Furthermore, the process of obtaining the feature contribution is as follows: Using candidate feature data and global label data as input, under the federated learning encryption framework, the candidate feature data is divided into blocks and encrypted using Paillier encryption, coordinated with global labels and broadcast, and the encryption domains of candidate features and global labels are aligned based on unique user identifiers. Nodes of the distributed data calculate the mutual information between a single feature and a global label in the encryption domain and sum them, then upload the encrypted value of the total mutual information sum. The encrypted sum of all distributed data nodes is aggregated and decrypted, and the proportion of the mutual information sum of a single distributed data node to the global sum is calculated, which is the feature contribution. Finally, it is verified that the sum of the contributions of all distributed data nodes is 1, and the feature contribution is output.

[0010] Furthermore, the process of outputting a high-quality feature set is as follows: Based on the federated learning privacy protection framework, this approach takes the discriminative features and quantified feature contributions selected locally by each node as input. It binds the local discriminative features with their own feature contributions, and verifies the association between features and global labels based on unique user identifiers to ensure data consistency. Then, it performs Paillier block encryption on the bound discriminative features. Within the local encryption domain, the encrypted feature value is multiplied by the contribution to obtain node-level encrypted weighted features. Each node uploads these features to collaborating nodes, which use the additive homomorphism property to sum them according to user identifiers, resulting in a globally encrypted high-quality feature set. After decryption, low-variance features are removed, the format is standardized, and the association strength between features and global labels is verified, ultimately outputting a high-quality feature set.

[0011] Furthermore, the training process for the average model is as follows: Each node associates high-quality features with local business labels to form a training set; collaborative nodes deploy the initial model and broadcast parameters, loading initial values ​​of node contribution; nodes are trained using Mini-batchSGD, combined with Dropout and L2 regularization to prevent overfitting, and gradients, data quality, and training efficiency are recorded synchronously; collaborative nodes calculate gradient bias to determine Byzantine behavior, penalize updated contribution, and filter out nodes with low contribution or high Byzantine probability; retained nodes encrypt gradients and upload them, collaborative nodes sum them by contribution weight and decrypt them to obtain the global gradient, update model parameters, and broadcast it for iteration. The iteration terminates when the loss change rate is less than a certain percentage for multiple consecutive rounds or when iterates for k rounds, and outputs a Byzantine fault-tolerant weighted federated average model.

[0012] Furthermore, the process of generating user profiles is as follows: Using a trained Byzantine fault-tolerant weighted federated average model as the core tool, this method extracts and standardizes multimodal data of target users by combining multimodal dimension dynamic weight operations, and associates it with historical profile metadata; calculates real-time dynamic weights and classifies dimension priorities; inputs the data into the model to infer a preliminary profile; adjusts and normalizes the data according to the weights to form a prototype profile; verifies its consistency with historical profiles and logical rationality; and finally outputs a structured user profile with quantitative indicators and qualitative labels.

[0013] A distributed user profile construction method based on big data includes the following steps: Step 1: Obtain multimodal dimension data, construct a dynamic weight and multi-constraint sharding system based on the multimodal dimension data, introduce an elastic adjustment mechanism to shard and store the multimodal dimension data, and obtain distributed data; Step 2: Obtain candidate feature data and global label data based on multimodal dimensional data. Calculate the sum and proportion of mutual information between candidate feature data and global label data to obtain the feature contribution. Perform local hierarchical filtering based on distributed data to obtain discriminative features. Weightedly fuse the discriminative features and feature contribution to output a high-quality feature set. Step 3: Based on the high-quality feature set, train the average model by improving the FedAvg algorithm, and output the user profile based on the trained average model combined with dynamic weights. The improved FedAvg algorithm introduces node contribution for gradient weighted updates on the basis of the federated averaging framework, and filters low-quality node gradients through Byzantine behavior penalty, replacing the average strategy in the federated averaging framework.

[0014] The present invention provides a distributed user profile construction method and system based on big data, which has the following beneficial effects: (1) This invention constructs a dynamic weight and multi-constraint sharding system through a multi-dimensional sharding storage module, combines an improved genetic algorithm to solve the optimal sharding scheme, and introduces an elastic adjustment mechanism to monitor node load, query latency and other indicators in real time, dynamically optimizes sharding allocation. Compared with the traditional fixed sharding mode, it can avoid data silos and node overload, ensure stable storage and efficient access of multimodal dimension data, adapt to the dynamic changes of user behavior and data popularity, and provide a highly available distributed data foundation for subsequent feature processing and profile construction.

[0015] (2) This invention uses Paillier homomorphic encryption technology to process candidate features, global labels and gradient data. Local feature screening and weighted fusion are both completed in the encrypted domain, and only the cooperating node holds the private key to decrypt. Feature contribution calculation is achieved through encrypted domain alignment and homomorphic summation, avoiding leakage of original data across nodes. It meets the privacy compliance requirements of finance, government and other fields, and activates the value of multi-source data while avoiding the risk of user privacy leakage.

[0016] (3) This invention improves the model's anti-interference ability by adopting an improved FedAvg algorithm, which uses node contribution weighted gradient updates and Byzantine behavior penalties to filter low-quality nodes. When outputting the profile, it combines real-time dynamic weights of multimodal dimensions, prioritizes high-importance dimensions, and corrects abnormal fluctuations through historical consistency and logical rationality checks. The generated structured profile has both global representativeness and dynamic adaptability, which can accurately support businesses such as precision marketing and intelligent risk control, and reduce enterprise operating costs. Attached Figure Description

[0017] Figure 1 This is a schematic diagram of the system of the present invention; Figure 2 This is a schematic diagram of the overall method of the present invention. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0019] Example 1: Please see Figure 1 Embodiment 1 of this application provides a distributed user profile construction system based on big data, the system comprising: The multi-dimensional sharding storage module acquires multi-modal dimensional data, constructs a dynamic weight and multi-constraint sharding system based on the multi-modal dimensional data, and introduces an elastic adjustment mechanism to shard and store the multi-modal dimensional data to obtain distributed data; Acquisition of multimodal dimensional data: Based on a distributed data acquisition system, five categories of data—identifier, spatial, attribute, behavioral, and temporal—are selectively extracted from the entire user interaction chain and business systems. After standardization processing, these data are transformed into multimodal dimensional data in a unified format. Identifier dimension data: Unique user IDs are obtained through the user registration system and login authentication module. After deduplication and verification, a unique set of user identifiers is formed to ensure data correlation. Spatial dimension data: Combining user IP address resolution, calling geographic information interfaces and the regional information filled in during registration, and standardizing the data according to the three levels of province, city, and district to form structured spatial data; Attribute dimension data: Static attributes such as age and gender are extracted from the user profile database and combined with the consumption level assessment results of the business system, that is, calculated based on historical transaction data, to form complete attribute data; Behavioral data: Real-time capture of user access records and interactive operations, such as user clicks, dwell time, and conversions, through a log collection system. Dynamic behavioral data is generated by statistically analyzing metrics such as access frequency and interaction depth within a fixed time window, such as the past 7 days. Time-dimensional data: Synchronized data generates timestamps and update records, which are extracted from database change logs and combined with query logs to statistically analyze query distribution over different time periods, forming time-characteristic data; The data of various dimensions are formatted uniformly, such as converting timestamps to standard time formats, standardizing regional names, and removing outliers, such as invalid IPs and filling in missing attributes. The final output is a multimodal dimensional dataset that can be directly used for dynamic weight calculation and sharding processing.

[0020] The process of constructing a dynamic weighted and multi-constraint partitioning system is as follows: Based on standardized multimodal data including identifiers, space, attributes, behaviors, and time, a multi-constraint segmentation system adapted to dynamic scenarios is constructed through dynamic weight adaptive calculation, multi-constraint segmentation value quantification, the use of an improved genetic algorithm to solve for the optimal solution, and elastic adjustment of the closed loop. The multimodal dimensional data is regularized and mapped to the corresponding attribute space. For example, the spatial dimension is mapped to the provincial, municipal and district hierarchical structure, and the behavioral dimension is mapped to the access frequency and interaction depth indicator set. Metadata such as query logs, data update records, node status data or bandwidth are extracted simultaneously. Based on the normalized dimensional data and metadata, the weights of the multimodal dimensional data at time t are calculated according to a preset dynamic weighting function. The query frequency and time interval of the dimensions are statistically analyzed using query logs to generate a query frequency decay term. The data popularity and update frequency are statistically analyzed using data update records and access volume to generate a data popularity gain term. The two results are weighted and fused using the balance coefficient determined by the analytic hierarchy process to obtain the dynamic weights of each dimension, reflecting the real-time importance of the dimensions. Based on dynamic weights and node status data, a sharding value function is constructed that includes load balancing, query latency, migration cost, and computing power fluctuation stability. Load balancing is calculated based on the ratio of node request volume to computing power; query latency is calculated by combining physical distance, bandwidth, and request volume; migration cost is calculated by shard size, migration times, and unit cost; and stability is calculated based on the fluctuation of computing power utilization within a time window. The entropy weight method is used to determine the weights of the four indicators, and the weighted average is used to obtain the comprehensive value of the sharding scheme. By improving the genetic algorithm to solve the optimal partitioning, the algorithm adapts to the requirements of dynamic weights and multi-index evaluation on the basis of the traditional genetic algorithm framework, thereby improving the solution efficiency and adaptability of the optimal partitioning scheme. It quickly finds the optimal partitioning scheme with multiple constraints, such as load balancing, low latency, low migration cost and high computing power stability, that is, it determines the partitioning threshold and node allocation strategy for each modality dimension. The algorithm uses a pre-defined combination of dimensional sharding thresholds as the population of individuals, such as sharding an individual based on city-level space or sharding based on 7-day access frequency. It abandons traditional single-indicator approaches, such as load balancing alone, and uses the reciprocal of a multi-constraint sharding value function as the fitness, ensuring optimal overall performance. Crossover operations focus on dimensional threshold exchange, preserving high-quality sharding logic, while mutation operations fine-tune dimensional weights or thresholds to avoid local optima. Combining convergence performance, a fitness change rate of <1% for 5 consecutive generations, and an iteration limit (e.g., setting the limit to 100 generations), the algorithm balances accuracy and efficiency. Using the combination of dimensional sharding thresholds as the population, it initializes and generates multi-group sharding schemes. The reciprocal of the sharding value function is used as the fitness function, and iterative optimization is achieved through genetic operations such as roulette wheel selection, dimensional threshold crossover, and weight or threshold mutation. When the fitness change rate for 5 consecutive generations is <1% or the iteration reaches 100 generations, the optimal sharding scheme is output, determining the sharding thresholds for each dimension and the node allocation strategy. The system monitors the sharding system's operational status in real time, collecting load balancing and query latency data and comparing them with target thresholds. When any indicator fails to meet the target, it dynamically calculates the number of new shards based on the difference between the current indicator and the target value, and re-executes the weight calculation and solution process to achieve adaptive adjustment of the sharding system, ensuring continuous adaptation to dynamic data changes and node load fluctuations.

[0021] The process of sharding and storing multimodal dimensional data is as follows: Using standardized multimodal dimensional data as the processing object, relying on dynamic weights and multi-constraint optimal sharding schemes, and adjusting the logic with an elastic adjustment mechanism, data sharding and distributed storage are completed to form a dynamically adaptable distributed dataset. Based on the optimal sharding scheme obtained by the improved genetic algorithm, the sharding thresholds for each modality dimension are clearly defined. For example, the spatial dimension is sharded by city, and the behavioral dimension is sharded by access frequency in the past 7 days, along with node allocation rules. The logic of splitting by dimension thresholds and matching data ownership is used to split multimodal dimension data into several independent shards. Each shard is bound to a corresponding distributed node, and matching is performed based on node computing power, bandwidth, and shard distance. The split data shards are synchronized to the assigned target nodes and stored using a distributed file system. Simultaneously, shard metadata is recorded, including shard size, dimensional composition, storage node, and update timestamp, forming the initial distributed data storage architecture. The load balance of each node is continuously monitored through the metadata collection module, i.e., the ratio of node request volume to computing power, query latency including transmission latency and processing latency, as well as dynamic indicators such as the update frequency and query popularity of shard data, and compared in real time with preset target thresholds, such as a load balance of 92% and a query latency of 85ms. When monitoring metrics fail to reach thresholds, such as load balancing < 92% or query latency > 85ms, adjustments are automatically triggered. Based on the difference between the current metrics and target values, the number of new shards is dynamically calculated, dynamic weight calculation and improved genetic algorithm are re-executed to generate an adjusted sharding scheme. Shard splitting and merging are completed according to the new scheme, nodes are redistributed, and shard metadata is updated synchronously to avoid excessive resource consumption during data migration. After the adjustment is completed, the load, query latency, and other metrics of each node are verified to meet the standards, ensuring stable data storage and efficient access of sharded data, ultimately forming distributed data adapted to dynamic scenarios.

[0022] The privacy protection module acquires candidate feature data and global label data based on multimodal dimensional data. It calculates the sum and proportion of mutual information between candidate feature data and global label data to obtain the feature contribution. It performs local hierarchical screening based on distributed data to obtain discriminative features. It then weights and fuses the discriminative features with the feature contribution to output a high-quality feature set. Acquisition of candidate feature data and global label data: The system extracts multimodal dimension data under its jurisdiction from the storage modules of each distributed data source, including identifier, spatial, attribute, behavioral, and temporal dimensions, as the original data source for feature and label generation, ensuring that the data covers all dimensions of user information. Based on a sliding window mechanism with a window size of 1000 data points, the mean and standard deviation of each feature within the window are calculated in real time to standardize the original feature data, avoiding feature distortion caused by dynamic changes in data distribution, and obtaining standardized local features. Based on pre-acquired local business tags, such as high-consumption users and high-frequency access users, the system calculates the mutual information between standardized local features and local business tags to measure the correlation strength. Combined with a dynamic threshold adjusted based on the ratio of node users to global users, feature data with mutual information higher than the dynamic threshold is selected to form candidate feature data, filtering out redundant features. Each distributed data node performs Paillier homomorphic encryption on its local business tags to prevent leakage of original tag information. The encrypted tag data is then uploaded to the distributed data nodes. The collaborative distributed data node receives the encrypted tag data from all nodes and uses the Paillier additive homomorphic property to summarize and statistically analyze the tags within the encrypted domain, generating a global tag set. After decryption, only the collaborative nodes holding the private key obtain the structured global tag data, which contains unified business tags for all users. The format of the candidate feature data, such as numerical type and dimension name, is standardized and verified one-to-one with the user identifier of the global tag data to ensure consistency between the two. Finally, the standardized candidate feature data and global tag data are output.

[0023] The process of obtaining feature contribution is as follows: Using standardized candidate feature data and encrypted global label data as core inputs, mutual information calculation, summation statistics and proportion quantification are completed under the federated learning encryption framework, ensuring data privacy and security throughout the process, and finally obtaining the feature contribution of each node. Each node performs Paillier block encryption on the candidate feature data, and the collaborating node encrypts the aggregated global label data and broadcasts it to all nodes. Simultaneously, based on unique user identifiers, the encrypted domains of candidate features and global labels are aligned to ensure consistency in the association between users, features, and labels, avoiding data misalignment. In the encrypted state, each node calculates the joint probability, feature marginal probability, and label marginal probability of each candidate feature value and each global label value. The joint probability is summed using the Paillier additive homomorphic property and then divided by the number of global users. Through Taylor expansion and approximate logarithmic operations, the logarithm is transformed into a polynomial calculation to solve for the encrypted mutual information value between a single candidate feature and the global label, measuring the global association strength of a single feature. Each node performs its own... The encrypted values ​​of mutual information of candidate features are summed to obtain the total encrypted value of mutual information between the candidate feature set of this node and the global label, and then uploaded synchronously to the collaborating node. The collaborating node receives the encrypted value of the total mutual information of all nodes, uses the Paillier homomorphic property to perform summation within the encrypted domain, and obtains the global encrypted value of the total mutual information of all node candidate features and the global label. The collaborating node decrypts the node mutual information sum and the global mutual information sum. Only the collaborating node holds the private key and calculates the proportion of the mutual information sum of a single node to the global mutual information sum. This proportion is the feature contribution of the node, reflecting the overall explanatory power of the node's candidate features to the global label. The feature contribution of all nodes is verified to be 1 to ensure that the calculation logic is without deviation. Finally, the feature contribution is output.

[0024] The process of obtaining discriminative features through local hierarchical screening is as follows: By preprocessing and filtering distributed data and performing core discriminative screening, feature selection is completed locally on each node, ultimately yielding highly discriminative local features. Distributed data is extracted from the storage module, and firstly, adaptive standardization using a sliding window is applied to eliminate dimensional differences and avoid feature distortion caused by dynamic changes in data distribution. Then, invalid features with missing values ​​exceeding 30% or outliers are removed, initially filtering redundant data. The variance of each preprocessed feature is calculated, and features with variances below a preset threshold are removed. Based on the global feature variance distribution, the top 80% of features with the highest variance are retained; these are low-fluctuation features with small value differences, which are beneficial for users. The discriminative contribution of user profiles is weak, and after filtering, a preliminary effective feature set is obtained. Using local business tags, such as high-spending users and high-frequency users, as a reference, the mutual information between the preliminary effective features and local business tags is calculated to measure the correlation strength between features and tags. A dynamic threshold is adopted, which is adjusted based on the ratio of the number of users in a node to the total number of users in the world. The smaller the node size, the higher the threshold. Features with mutual information higher than the threshold are selected to ensure that the retained features have strong discriminative power. The format of the selected features is standardized to verify the completeness of the association between features and user identifiers and to avoid data misalignment. Finally, the discriminative features after local hierarchical filtering are obtained.

[0025] The process of outputting a high-quality feature set is as follows: Based on the federated learning privacy protection framework, the discriminative features and quantified feature contributions after local hierarchical screening at each node are used as core inputs. Through encrypted domain weighted calculation, global aggregation and post-processing, a high-quality feature set with both global representativeness and privacy security is output. Each distributed node binds the discriminative features obtained from local hierarchical filtering with its corresponding feature contribution based on the mutual information ratio between the node's candidate features and the global label. At the same time, it performs association verification between features and global labels based on a unique user identifier to ensure the consistency of features, contribution, and users and avoid data misalignment. Each node performs Paillier block encryption on the bound discriminative features to maintain the encrypted state of the features to ensure privacy and security and prevent the leakage of original feature information during cross-node transmission, generating an encrypted discriminative feature set. Each node, within its local encryption domain, multiplies the encrypted value of each discriminative feature with its own feature contribution using the Paillier multiplicative homomorphic property to obtain a node-level encrypted weighted feature. Nodes with higher contributions have a higher weight in the global fusion, achieving differentiated fusion where high-quality features contribute more and inefficient features have a smaller proportion. Each node uploads its encrypted weighted feature to a collaborating node. The collaborating node, using the Paillier addition homomorphic property, sums the encrypted weighted features of all nodes according to user identifiers to obtain a global set of high-quality encrypted features. In other words, each user's global feature is the sum of the weighted features of all nodes. The process involves several steps: First, a high-quality feature set is decrypted using a private key by the coordinating nodes to obtain the original global feature set. Then, post-processing optimization is performed. This involves removing low-value features with variance below 50% of the global mean, eliminating redundant features, standardizing feature formats (i.e., standardizing numerical types and dimension names), verifying the correlation strength between feature dimensions and global labels, and retaining features with mutual information higher than the global average. After post-processing verification, the final high-quality feature set is output. This set integrates the core value of the discriminative features from each node and avoids the drawback of weighting based solely on node size through contribution weighting, while ensuring privacy and security throughout the process.

[0026] The offline collaboration module trains an average model based on a high-quality feature set by improving the FedAvg algorithm, and outputs a user profile based on the trained average model combined with dynamic weights. Among them, the improved FedAvg algorithm introduces node contribution for gradient weighting updates on the basis of the federated averaging framework, and filters low-quality node gradients through Byzantine behavior penalty, replacing the average strategy in the federated averaging framework. Using a high-quality feature set optimized by federated privacy fusion and post-processing as the training data basis, and relying on the improved FedAvg algorithm, the average model is robustly trained by weighting by Byzantine fault tolerance and node contribution, following a federated learning process of distributed local training, global gradient collaboration, and model iterative optimization. S301: Training preparation and model initialization; Each distributed node obtains a global high-quality feature set from the collaborating node, associates it with its own stored business tags according to the local user identifier, such as high-consumption users or high-frequency access users, to form a local training dataset of high-quality features and tags, ensuring data alignment without misalignment; The collaborating node deploys machine learning models, such as logistic regression and lightweight neural networks, and broadcasts the initial model parameters, weights, and biases to all distributed nodes to ensure that the training starting point of each node is consistent; At the same time, the initial values ​​of node contribution are loaded; S302: Each node uses a local high-quality feature and label dataset as input and employs mini-batch stochastic gradient descent for local training. The number of training epochs and batch size are set, such as 10 training epochs and 64 batches pre-set. During training, the quality of local data is calculated in real time, and the label accuracy and training efficiency are monitored through the validation set. The training time per epoch is recorded synchronously, and the gradient data of the local model, i.e., the parameter update amount in each epoch, is recorded to provide a basis for subsequent contribution adjustment. Local training uses Dropout regularization with a dropout rate of 0.2 and L2 weight decay with a coefficient of 0.001 to ensure the generalization ability of the local model. S303: Each node uploads its local training gradients to the collaborating node. The collaborating node calculates the L2 deviation between the gradient of a single node and the mean gradient of all nodes to measure gradient consistency. If the deviation exceeds 0.2, it is judged as potential Byzantine behavior. Combining the initial contribution (data quality × 0.7 + training efficiency × 0.3) and the probability of Byzantine behavior, the gradient is dynamically updated according to the gradient deviation. The larger the deviation, the higher the probability. The node contribution is penalized and adjusted. Its contribution = initial contribution × (data quality / target quality 0.95) × (1 - 0.8 × Byzantine behavior probability). If the node contribution ≤ 0 or the probability of Byzantine behavior ≥ 0.5, the gradient of that node is temporarily excluded from global fusion to avoid malicious or low-quality gradients interfering with model training. S304: The retained nodes perform Paillier block encryption on their local gradients to ensure the privacy and security of gradient data during cross-node transmission and avoid model privacy risks caused by gradient leakage; each node uploads the encrypted gradients to the collaborating nodes, which use the homomorphic property of Paillier addition to calculate the global gradient by weighting and summing according to the node's contribution, where global gradient = Σ(node ​​contribution × node encrypted gradient). The higher the contribution of a node, the higher its gradient accounts for in the global fusion, replacing the simple averaging strategy of traditional FedAvg; the collaborating nodes decrypt the global weighted gradients with their private keys to obtain the original global gradients that can be used for model updates. S305: The collaborative node updates the global model parameters based on the decrypted global gradient and the preset learning rate. The new parameter = current parameter - learning rate × global gradient, ensuring that the model iterates in the direction of loss reduction. The updated global model parameters are then rebroadcast to all remaining nodes. Each node receives the updated global model parameters and overwrites its local model parameters, then enters the next round of local training. The process of local training, contribution adjustment, gradient fusion and parameter update is repeated until the termination condition is met. S306: When the change rate of the validation set loss of the global model is less than 1% for 5 consecutive rounds, convergence is determined, or the number of iterations reaches 100 rounds, i.e. the iteration limit, and training is stopped; the collaborative nodes save the final global model parameters to form the average model after training, i.e. the Byzantine Fault-Tolerant Weighted Federated Average Model BFT-W-FedAvg. This model integrates the training value of high-quality features of each node and improves robustness through contribution weighting and Byzantine fault tolerance mechanism.

[0027] The process of generating user profiles is as follows: Using the trained Byzantine fault-tolerant weighted federated average model as an inference tool, combined with multimodal dynamic weights to reflect the importance of dimensions, and following the logic of data input, dynamic weight adaptation, model inference, profile fusion and verification output, accurate user profiles adapted to dynamic scenarios are generated. Extract full-volume multimodal dimension data of the target user, including identifier, spatial, attribute, behavioral and temporal dimensions. Adaptively standardize the data through a sliding window to be consistent with the standards of the feature engineering stage, eliminate differences in units, and ensure that the data format is aligned with the high-quality feature set input for model training. Associate the user's historical profile metadata, such as the last profile generation time and core tags, to provide a reference for dynamic weight adaptation and profile updates, and avoid duplicate calculations and data misalignment. Based on the current query logs, data update frequency, and user activity, the dynamic weights of each dimension of the multimodal model are recalculated. The calculation logic for query frequency decay and data popularity gain in Module 1 is the same, resulting in a real-time dimension weight set, such as a weight of 0.4 for the recent behavior dimension, 0.2 for the spatial dimension, and 0.15 for the attribute dimension. The dimensions are prioritized according to their weight ratio. The top 3 dimensions with the highest weights, such as behavior, time, and space, are the core influencing dimensions for the user profile, while the rest are auxiliary dimensions, ensuring that the user profile focuses on the user's recent key characteristics. The standardized user multimodal data is input into the trained BFT-W-FedAvg model. The model calculates through forward propagation and outputs preliminary quantitative results for each user profile dimension, such as a consumption potential score of 0.85, a high-frequency access activity level, and interest tags such as digital products and outdoor sports. During the inference process, the model automatically filters out the influence of abnormal features and relies on the Byzantine fault tolerance mechanism to ensure the stability and rationality of the preliminary profile results. The preliminary profile results are weighted and adjusted according to the real-time dimension weight set: the quantitative results of the core influencing dimensions are multiplied by the weight amplification coefficient, such as multiplying the behavior dimension with a weight of 0.4 by 1.2, and the quantitative results of the auxiliary dimensions are multiplied by the weight adaptation coefficient, such as multiplying the attribute dimension with a weight of 0.15 by 0.9, to balance the influence of dynamic and static features; the weighted profile indicators are normalized and mapped to the [0,1] interval, and integrated to form a preliminary structured profile of basic attributes, behavioral preferences, consumption potential and interest tags. Among them, the higher the proportion of dynamic weight, the higher the display priority and quantitative weight in the profile. Compare the core tags of the initial user profile with those of the historical profile. For example, if the consumption level changes from mid-range to high-end, and the difference stems from changes in the dimensional features corresponding to dynamic weights, such as recent high-consumption behavior, then the adjustment is retained. If the difference is due to abnormal fluctuations, such as a single erroneous operation, then it is smoothly corrected by combining historical data. Filter contradictory user profile indicators through business rules, such as the incompatibility of high-frequency access and low-activity scores, to ensure the consistency of the user profile logic. After verification, output a structured final user profile, including quantitative indicators, such as consumption potential 0.82 and access frequency 0.78, and qualitative tags, such as high-value user, digital enthusiast, and local consumption preference.

[0028] Example 2: Please see Figure 2 Based on Example 1, Example 2 of this application also provides a distributed user profile construction method based on big data, including the following specific steps: Step 1: Obtain multimodal dimension data, construct a dynamic weight and multi-constraint sharding system based on the multimodal dimension data, introduce an elastic adjustment mechanism to shard and store the multimodal dimension data, and obtain distributed data; Step 2: Obtain candidate feature data and global label data based on multimodal dimensional data. Calculate the sum and proportion of mutual information between candidate feature data and global label data to obtain the feature contribution. Perform local hierarchical filtering based on distributed data to obtain discriminative features. Weightedly fuse the discriminative features and feature contribution to output a high-quality feature set. Step 3: Based on the high-quality feature set, train the average model by improving the FedAvg algorithm, and output the user profile based on the trained average model combined with dynamic weights. The improved FedAvg algorithm introduces node contribution for gradient weighted updates on the basis of the federated averaging framework, and filters low-quality node gradients through Byzantine behavior penalty, replacing the average strategy in the federated averaging framework.

[0029] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented in software, the above embodiments can be implemented, in whole or in part, as a computer program product. Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution.

[0030] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0031] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A distributed user profile construction system based on big data, characterized in that, The system includes: The multi-dimensional sharding storage module acquires multi-modal dimensional data, constructs a dynamic weight and multi-constraint sharding system based on the multi-modal dimensional data, and introduces an elastic adjustment mechanism to shard and store the multi-modal dimensional data to obtain distributed data; The privacy protection module acquires candidate feature data and global label data based on multimodal dimensional data. It calculates the sum and proportion of mutual information between candidate feature data and global label data to obtain the feature contribution. It performs local hierarchical screening based on distributed data to obtain discriminative features. It then weights and fuses the discriminative features with the feature contribution to output a high-quality feature set. The offline collaboration module trains an average model based on a high-quality feature set by improving the FedAvg algorithm, and outputs a user profile based on the trained average model combined with dynamic weights. Among them, the improved FedAvg algorithm introduces node contribution for gradient weighting updates on the basis of the federated averaging framework, and filters low-quality node gradients through Byzantine behavior penalty, replacing the average strategy in the federated averaging framework.

2. The distributed user profile construction system based on big data according to claim 1, characterized in that, The process of acquiring multimodal dimensional data is as follows: Through a distributed data acquisition system, five categories of data—identity, space, attribute, behavior, and time—are selectively captured from the entire user interaction chain and business systems. After standardization processing, these data are transformed into multimodal dimensional data in a unified format.

3. The distributed user profile construction system based on big data according to claim 2, characterized in that, The process of constructing a dynamic weighted and multi-constraint partitioning system is as follows: Multimodal dimensional data is normalized and metadata is extracted. Based on the normalized multimodal dimensional data and metadata, the weights of the multimodal dimensional data at time t are calculated according to a preset dynamic weight function. The dynamic weights of each dimension are obtained by weighted fusion using the balance coefficients determined by the analytic hierarchy process. A sharding value function is constructed based on dynamic weights and metadata. The entropy weight method is used to determine the index weights, and the comprehensive value of the sharding scheme is obtained by weighting. Based on comprehensive value, a multi-group partitioning scheme is generated by using a preset combination of dimensional partitioning thresholds as individuals in the population. The inverse of the partitioning value function is used as the fitness function, and the optimal partitioning scheme is output through iterative optimization by roulette wheel selection, dimensional threshold crossover, weight or threshold mutation genetic operations.

4. The distributed user profile construction system based on big data according to claim 3, characterized in that, The process of acquiring distributed data is as follows: Using standardized multimodal dimensional data as the processing object, an improved genetic algorithm is introduced based on dynamic weights to solve a multi-constraint optimal sharding scheme. First, the multi-constraint optimal sharding scheme is implemented to split the data and bind it to distributed nodes, initial storage shards are created and metadata is recorded. Then, the load of distributed nodes, query latency and dynamic indicators of initial storage shards are monitored. If the initial storage shards do not reach the preset threshold, the initial storage shards are dynamically adjusted. Finally, the dynamic indicators are verified to meet the standards, forming distributed data adapted to dynamic scenarios.

5. The distributed user profile construction system based on big data according to claim 1, characterized in that, The process of obtaining candidate feature data and global label data is as follows: Multimodal dimension data under jurisdiction is extracted from distributed data, and standardized local features are obtained after adaptive standardization through a sliding window. Based on the pre-acquired local business labels, the mutual information between the standardized local features and the local business labels is calculated. A dynamic threshold is obtained based on the number of users, and candidate feature data is obtained by combining the mutual information and the dynamic threshold. Local business tags are encrypted and uploaded via Paillier using distributed data. The encrypted tags are then aggregated and decrypted using the additive homomorphic property of the distributed data to obtain global tag data.

6. The distributed user profile construction system based on big data according to claim 5, characterized in that, The process of obtaining feature contribution is as follows: Using candidate feature data and global label data as input, under the federated learning encryption framework, the candidate feature data is divided into blocks and encrypted using Paillier encryption, coordinated with global labels and broadcast, and the encryption domains of candidate features and global labels are aligned based on unique user identifiers. Nodes of the distributed data calculate the mutual information between a single feature and a global label in the encryption domain and sum them, then upload the encrypted value of the total mutual information sum. The encrypted sum of all distributed data nodes is aggregated and decrypted, and the proportion of the mutual information sum of a single distributed data node to the global sum is calculated, which is the feature contribution. Finally, it is verified that the sum of the contributions of all distributed data nodes is 1, and the feature contribution is output.

7. A distributed user profile construction system based on big data according to claim 6, characterized in that, The process of outputting a high-quality feature set is as follows: Based on the federated learning privacy protection framework, the system takes the discriminative features and quantified feature contributions selected locally by each node as input. It binds the local discriminative features with their own feature contributions, and verifies the association between features and global labels based on unique user identifiers to ensure data consistency. Then, it performs Paillier block encryption on the bound discriminative features. Within the local encryption domain, the encrypted feature value is multiplied by the contribution to obtain the node-level encrypted weighted feature. Each node uploads this feature to the collaborating node, which uses the additive homomorphic property to sum the features according to the user identifier, obtaining a globally encrypted high-quality feature set. After decryption, low-variance features are removed, the format is standardized, and the correlation strength between the features and the global labels is verified, ultimately outputting a high-quality feature set.

8. The distributed user profile construction system based on big data according to claim 1, characterized in that, The training process for the average model is as follows: Each node associates high-quality features with local business labels to form a training set; collaborative nodes deploy the initial model and broadcast parameters, loading initial values ​​of node contribution; nodes are trained using Mini-batchSGD, combined with Dropout and L2 regularization to prevent overfitting, and gradients, data quality, and training efficiency are recorded synchronously; collaborative nodes calculate gradient bias to determine Byzantine behavior, penalize updated contribution, and filter out nodes with low contribution or high Byzantine probability; retained nodes encrypt gradients and upload them, collaborative nodes sum them by contribution weight and decrypt them to obtain the global gradient, update model parameters, and broadcast it for iteration. The iteration terminates when the loss change rate is less than a certain percentage for multiple consecutive rounds or when iterates for k rounds, and outputs a Byzantine fault-tolerant weighted federated average model.

9. A distributed user profile construction system based on big data according to claim 1, characterized in that, The process of generating user profiles is as follows: Using a trained Byzantine fault-tolerant weighted federated average model as the core tool, multimodal data of target users is extracted and standardized by combining multimodal dimension dynamic weight operations, and linked with historical profile metadata; real-time dynamic weights are calculated and dimension priorities are divided; the data is input into the model to infer a preliminary profile; and the profile is formed by weighted adjustment and normalization. Verify its consistency and logical rationality with historical profiles; finally, output a structured user profile with quantitative indicators and qualitative labels.

10. A distributed user profile construction method based on big data, characterized in that, Includes the following steps: Step 1: Obtain multimodal dimension data, construct a dynamic weight and multi-constraint sharding system based on the multimodal dimension data, introduce an elastic adjustment mechanism to shard and store the multimodal dimension data, and obtain distributed data; Step 2: Obtain candidate feature data and global label data based on multimodal dimensional data. Calculate the sum and proportion of mutual information between candidate feature data and global label data to obtain the feature contribution. Perform local hierarchical filtering based on distributed data to obtain discriminative features. Weightedly fuse the discriminative features and feature contribution to output a high-quality feature set. Step 3: Based on the high-quality feature set, train the average model by improving the FedAvg algorithm, and output the user profile based on the trained average model combined with dynamic weights. The improved FedAvg algorithm introduces node contribution for gradient weighted updates on the basis of the federated averaging framework, and filters low-quality node gradients through Byzantine behavior penalty, replacing the average strategy in the federated averaging framework.

Images