Data processing method and system based on game platform

Abstract

The invention provides a data processing method based on a game platform. Belongs to the technical field of data processing. The method comprises the steps of multi-source data acquisition and digital twin modeling; constructing a real-time data stream processing environment; kafka is used for receiving a real-time data stream, the real-time data stream is stored in a distributed file system or a column database, and a stream processing engine is used for carrying out instant analysis on the real-time data stream; performing batch processing and analysis on the historical data stored in the HDFS or the HBase by using a batch processing tool, and mining potential data modes and trends; and carrying out feature engineering processing. And aggregating the local models of the participants through a federal learning platform. Through combination of batch processing and stream processing, quick response of real-time data is ensured, deep mining of historical data is realized, and complementary advantages are formed.

Description

Technical Field

[0001] The invention proposes a data processing method and system based on a game platform, belonging to the technical field of data processing. Background Art

[0002] In the field of game processing, traditional data processing methods often face the problem that stream processing (such as Flink) and batch processing (such as Spark) systems run independently, resulting in the inability to guarantee the consistency of data at different time points; therefore, there is an urgent need for a new method that can process game data efficiently and intelligently to break the routine and meet the growing demand for game data processing. Summary of the invention

[0003] The present invention provides a data processing method and system based on a game platform to solve the problems mentioned in the above background technology: The present invention proposes a data processing method based on a game platform, the method comprising: S1. Collect multi-source data; S2. Build a real-time data stream processing environment; use Kafka to receive real-time data streams and store them in a distributed file system or columnar database; use the stream processing engine to perform instant analysis on real-time data streams and extract valuable information; use batch processing tools to batch process and analyze historical data stored in HDFS or HBase to mine potential data patterns and trends; S3, perform feature engineering processing; S4. Aggregate the local models of each participant through the federated learning platform.

[0004] The data processing system based on a game platform proposed in the present invention includes a memory, a processor, and a computer program stored in and executable on the memory, and the processor executes the program to implement any of the data processing methods based on a game platform as described above.

[0005] Beneficial effects of the invention: The technical solution proposed in the invention receives real-time data streams through Kafka, which can ensure the immediacy and low latency of data, thereby improving the user's gaming experience and the overall fluency of the system; using a stream processing engine (such as Flink) to perform real-time analysis on real-time data streams can quickly extract valuable information, thereby improving the decisiveness and accuracy of decision-making; through the cooperation of Kafka and Flink, real-time data streams and historical data can be processed simultaneously, avoiding the limitation of traditional methods that can only process one type of data, and improving the comprehensiveness of data processing; storing real-time data streams in distributed file systems (such as HDFS) or columnar databases (such as HBase) not only ensures the large-scale storage capacity of data, but also provides good scalability and fault tolerance, and can also effectively To cope with the subsequent large-scale growth of data; using batch processing tools (such as Spark) to batch process and analyze historical data stored in HDFS or HBase, it can deeply explore potential data patterns and trends, and provide an effective basis for the formulation of long-term operation strategies for game platforms; and improve the accuracy of user behavior predictions; through the combination of batch processing and stream processing, it not only ensures the rapid response of real-time data, but also realizes the in-depth mining of historical data, forming complementary advantages; the construction of real-time data stream processing environment and batch processing environment can flexibly allocate resources according to actual needs. During peak periods, the resource requirements of real-time data stream processing can be prioritized; during trough periods, more resources can be used for batch processing and analysis; through reasonable resource scheduling and optimization, the overall operating cost can be reduced and resource utilization can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] Figure 1 This is a step diagram of the method described in the present invention. DETAILED DESCRIPTION

[0007] The preferred embodiments of the present invention are described below in conjunction with the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0008] One embodiment of the present invention, as Figure 1 As shown, a data processing method based on a game platform, the method comprising: S1. Obtain raw data from multiple data sources (e.g., game logs, player behavior records, sensor data), including structured data (e.g., player level, recharge records) and unstructured data (e.g., social chat records, voice commands); preprocess the collected raw data; apply feature extraction algorithms (e.g., PCA and LDA) to extract key features from the preprocessed data, the key features are used to reflect key information such as player activity, social influence, etc.; use 3D modeling software (e.g., Unity, UnrealEngine) to build a three-dimensional model of the game world, and combine the extracted feature data to build a digital twin model of the game world; verify and optimize the digital twin model by comparing the simulation results with the real game data; S2. Install and configure a stream processing framework (such as Apache Kafka and Apache Flink) and build a real-time data stream processing environment. Use Kafka to receive real-time data streams and store them in a distributed file system (such as HDFS) or a columnar database (such as HBase). Use a stream processing engine (such as Flink) to perform instant analysis on real-time data streams and extract valuable information (such as real-time player behavior patterns and game anomaly detection). Use batch processing tools (such as Spark SQL) to batch process and analyze historical data stored in HDFS or HBase to mine potential data patterns and trends. S3. Select the most representative features (such as player behavior features and game scene features) based on business needs and data characteristics; apply feature transformation algorithms to preprocess the features, generate new features through feature cross-pollination, and enrich the feature space; and divide the preprocessed feature data into training sets, validation sets, and test sets according to a certain ratio; S4. Initialize the global model on the federated learning platform, for example, through the TensorFlowFederated (TFF) framework; each participant (such as game server, player device, etc.) uses local data to train the local model and uploads the model parameters to the federated learning platform; the federated learning platform aggregates the local models of each participant and updates the global model; gradually optimize the performance of the global model through multiple iterative training; use the validation set to validate the global model and evaluate the performance of the model. Tune the model based on the validation results; The working principle and effect of the above technical solution are as follows: by collecting multiple data sources, the behaviors and interactions of players in the game can be fully captured; the combination of structured and unstructured data collected can provide a deeper understanding of players' behavior patterns, social influence and other potential in-game activities, thereby improving the breadth and depth of the data; by combining the stream processing framework with batch processing tools, it is possible to capture the data stream in the game in real time, identify real-time behavior patterns and anomalies, and process historical data to discover trends and laws over a long period of time. Through the combination of the two, the system can respond quickly and accurately predict future behaviors, thereby improving the ability to capture data and reducing risks and costs to a certain extent; by using 3D modeling software to build a digital twin model of the game world, and combining it with the extracted key feature data, it can not only simulate the dynamic changes in the game environment, but also be used to verify and optimize the accuracy and realism of the model. Thereby improving the gaming experience and prediction capabilities; applying feature extraction, transformation, and cross-generation algorithms, it is possible to screen out the most representative features from massive amounts of data, enhance the expressiveness of the model, enrich the feature space through feature cross-fertilization and expansion, and provide more comprehensive and useful information for the training of machine learning models, thereby enhancing the expressiveness of the model and improving the model's prediction and generalization capabilities; using a federated learning platform for distributed training, it is possible to complete the training of a global model without centrally storing the player's personal data, thereby improving player privacy and data security, and at the same time, improving the ability to efficiently optimize large-scale models in a distributed environment; through multiple iterations and optimization of global model training, it is possible to gradually improve the prediction accuracy and responsiveness of the game system, and when using the validation set for performance evaluation, it is possible to quickly adjust the model parameters and algorithm design, making the final model more accurate and efficient.

[0009] In one embodiment of the present invention, the S1 includes: S11. Identify multiple types of data sources (such as game logs, player behavior records, and sensor data), and select appropriate access methods based on the data source type; for example, obtain game logs through the backend; perform preliminary cleaning of the data source to remove invalid, duplicate, or abnormal data; integrate the cleaned data sources to form a unified data warehouse; S12: Standardize and normalize structured data (such as player levels and recharge records); perform text analysis, voice-to-text, sentiment analysis, etc. on unstructured data (such as social chat records and voice commands) to extract valuable information; the valuable information includes operating habits and preferences, for example, players frequently use voice commands such as "play background music" or "adjust screen brightness"; S13, applying PCA (principal component analysis) to extract key features, where the key features are used to reflect key information, including player activity and social influence; S14. Combine business needs and data characteristics, use filtering, wrapping or embedded feature selection methods to select the most representative features; use 3D modeling software such as Unity or UnrealEngine to build a three-dimensional model of the game world, including scenes, characters and props; S15. Integrate the extracted feature data into the three-dimensional model, construct a digital twin model of the game world, and synchronize virtual and real worlds. Verify the digital twin model by comparing the simulation results with the real game data, adjust the model parameters according to the verification results, and optimize the model performance.

[0010] The working principle and effect of the above technical solution are as follows: by identifying and integrating multiple types of data sources, it is possible to fully obtain the player's behavior, interaction and environmental information in the game, which is helpful to subsequently improve the player experience and enhance the game data analysis and decision-making capabilities; using different access methods and cleaning methods for different data sources can ensure data quality and consistency, and ultimately form a unified data warehouse, which helps to improve data utilization efficiency and system flexibility; standardizing, normalizing, and text analyzing structured and unstructured data respectively, so that various types of data can be compared and analyzed under a unified framework, improving the readability and operability of the data, and also extracting valuable information through techniques such as sentiment analysis. Information provides effective support for player behavior modeling and game experience optimization; by applying feature extraction methods such as PCA, key information can be screened out from massive data, which can accurately reflect the characteristics of player behavior and social networks, providing a more accurate basis for subsequent analysis and decision-making; combining business needs and data characteristics, different feature selection methods are used to effectively screen out the most representative features, improve the expressiveness and accuracy of the model, reduce data redundancy, improve the efficiency of the model, and reduce computational complexity and optimize system performance; by using 3D modeling software to build a three-dimensional model of the game world, and combining the extracted feature data to build a digital twin model, the dynamic environment in the game can be accurately simulated. The synchronization of virtual and reality can not only better predict and optimize the game experience, but also provide real-time feedback and adjustments to ensure the balance of the game system and the immersion of players; by performing virtual and real-time verification of the digital twin model, and adjusting the model parameters according to the verification results, the model performance can be continuously optimized to ensure its effectiveness in practical applications, improve the accuracy and stability of the model, and ensure the performance of the system in long-term use.

[0011] In one embodiment of the present invention, the S15 includes: S151, associating the extracted key feature data (such as player activity, social influence, etc.) with corresponding elements in the three-dimensional model (such as scene activity, character social network, etc.) through a mapping algorithm; S152. Based on the embedding representation method in deep learning, the feature data is embedded into the underlying structure of the three-dimensional model; this method can enable the model to incorporate more semantic information while maintaining the original geometric and physical properties, thereby improving the model's expression ability and prediction accuracy; S153. Capture and process real-time data from the real game world through a real-time data stream processing pipeline, where the real-time data is used to update the status and information in the digital twin model; build a simulation experiment environment, and place the digital twin model in it for simulation operation; evaluate the accuracy and reliability of the model by comparing the simulation results with the real game data, including comparing the player's behavior patterns, dynamic changes in the game world, etc.; S154. Based on the comparison results, adjust and optimize the parameters of the digital twin model, including adjusting the weight of feature data, optimizing the parameters of the synchronization algorithm, and improving the geometry and physical properties of the three-dimensional model. At the same time, considering the continuous evolution of the game world, the model needs to be updated and iterated regularly.

[0012] The working principle and effect of the above technical solution are as follows: by associating the extracted key feature data with the elements in the three-dimensional model and embedding these features into the underlying structure of the three-dimensional model through the embedding representation method in deep learning, the semantic expression ability of the model can be significantly enhanced, so that the model can not only maintain the original geometric and physical properties, but also integrate more in-depth information related to player behavior, game context, etc., thereby improving the accuracy and predictive ability of the model; by building a real-time data stream processing pipeline, data from the real game world can be captured and processed in real time, and the status and information in the digital twin model can be dynamically updated to ensure that the model always reflects the latest changes in the game environment and responds to the dynamic changes of player behavior and the game world in real time, thereby improving the adaptability and flexibility of the model; by placing the digital twin model By running the simulation in a simulated experimental environment and comparing it with real game data, the accuracy and reliability of the model can be effectively evaluated, potential problems in the model can be identified, and targeted adjustments and optimizations can be made to ensure that the model better predicts and reflects the player's behavior and the dynamic changes of the game world; according to the simulation comparison results, the parameters of the model are adjusted and optimized, including adjusting the weight of feature data, optimizing the parameters of the synchronization algorithm, and improving the geometric and physical properties of the three-dimensional model, which can continuously improve the performance of the model; the continuous optimization and regular iteration of the model can ensure that it still has high accuracy and high reliability in the process of the continuous evolution of the game world, ensuring that it can run stably for a long time; by combining deep learning and digital twin technology, a more personalized and intelligent gaming experience can be provided for each player. The model can automatically adjust the dynamic changes of the game world according to the player's behavioral characteristics and changes in the game environment, and enhance the player's immersion and participation.

[0013] In one embodiment of the present invention, the S151 includes: Before mapping, the extracted key feature data is preprocessed; and the feature data is analyzed for spatiotemporal characteristics, including time series analysis and spatial distribution analysis, to reveal the inherent laws and changing trends of the data; According to the spatiotemporal characteristics of the data and the structural characteristics of the 3D model, mapping rules are formulated; including determining which feature data should be mapped to which model elements, as well as the specific mapping method and parameter settings; based on the formulated mapping rules, the mapping algorithm is designed and implemented; The mapping results are presented in a visual manner, including marking the feature data in the 3D model in different colors, shapes or sizes; the accuracy and reliability of the mapping are verified by comparing the mapping results with the actual game data; if deviations or unreasonableness in the mapping results are found, adjustments and optimizations are made according to the verification results, including adjusting the mapping rules, optimizing the algorithm parameters or improving the structure of the 3D model.

[0014] The working principle of the above technical solution is as follows: by preprocessing the extracted key feature data and combining it with the analysis of spatiotemporal characteristics, the inherent laws and change trends of the data can be deeply revealed, which helps to more comprehensively understand the dynamic patterns and change mechanisms behind the data, thereby providing a more accurate basis for subsequent mapping work; by formulating accurate mapping rules based on the spatiotemporal characteristics of the data and the structural characteristics of the three-dimensional model, the correct association between the feature data and the model elements can be ensured; when formulating the rules, the data change laws and model characteristics are taken into consideration, which can ensure the rationality and efficiency of the mapping process; in addition, the design and implementation of the mapping algorithm helps to accurately embed the data into the three-dimensional model, thereby improving the model's ability to reflect the real world; by presenting the mapping results in a visual way, such as annotations of different colors, shapes or sizes, not only the readability and comprehensibility of the data are improved, but also the user's experience of interacting with the model is enhanced, and the cognition and understanding of the data and the model are improved; by comparing with real game data, the accuracy and reliability of the mapping results can be verified, and the deviations or unreasonableness in the mapping results can be effectively discovered, thereby providing a basis for further adjustment and optimization. The verification mechanism ensures that the system continues to develop in an accurate and reliable direction, and improves the model's ability to predict and respond to changes in reality; when there are deviations in the mapping results, the mapping rules can be adjusted in a timely manner, the algorithm parameters can be optimized, or the structure of the 3D model can be improved to improve the performance of the system. Such a continuous optimization mechanism can dynamically change according to different game environments and data to ensure that the mapping and model always maintain efficiency and adaptability; by optimizing the mapping process and improving the accuracy and interactivity of the model, the end-user experience will be significantly improved. Users can obtain clearer and more intuitive data feedback in the visual interface, and feel the system's real-time response to their behavior and the game environment, thereby increasing immersion and participation.

[0015] In one embodiment of the present invention, the S2 includes: S21. Select a stream processing framework (such as Apache Kafka and Apache Flink) based on business needs to build a real-time data stream processing environment; S22. Configure the Kafka cluster to ensure stable transmission and storage of real-time data streams, and set partition and replication strategies; use Kafka to receive real-time data streams and store them in a distributed storage system (such as HDFS or HBase) for persistent storage of data; S23. Use stream processing engines (such as Flink) to analyze real-time data streams in real time and extract valuable information (such as real-time player behavior patterns and game anomaly detection). At the same time, set up an alarm mechanism to monitor and warn of abnormal behaviors in real time. S24. Load historical data from the distributed storage system to provide a basis for batch processing; use batch processing tools (such as SparkSQL) to batch process and analyze historical data to explore potential data patterns and trends, such as player behavior patterns and changes in game popularity; at the same time, combine machine learning algorithms to predict and classify historical data.

[0016] The working principle and effect of the above technical solution are as follows: by introducing a stream processing framework (such as Apache Kafka and Apache Flink), efficient processing and analysis of real-time data streams can be achieved, thereby enhancing data quality and consistency; Kafka, as a high-throughput message queue system, can ensure fast and stable data transmission and reliably store data; and Flink can instantly analyze real-time data streams and extract valuable information, helping business parties to quickly respond to changes while ensuring real-time and accuracy; configuring a Kafka cluster and setting partition and replica strategies can improve the reliability and fault tolerance of data transmission and avoid data loss due to node failures. The use of distributed storage systems (such as HDFS or HBase) ensures persistent storage and high availability of data. The real-time analysis capability of the stream processing engine can detect and identify abnormal behaviors (such as cheating, game failures, or performance bottlenecks) in a timely manner, improving the gaming experience and transparency. After setting up the alarm mechanism, the system can automatically perform real-time monitoring and early warning to ensure that abnormal situations can be quickly discovered and handled, reducing potential risks. Loading historical data from the distributed storage system and combining it with batch processing tools can analyze large-scale historical data and explore the potential patterns and trends behind the data, reducing data processing costs and enhancing analytical insights into big data. It enhances the depth and breadth of data analysis; by combining machine learning algorithms with historical data, it can predict future data changes and optimize game content, player experience or marketing strategies; for example, based on the prediction results, the system can adjust the game content release plan, player recommendation mechanism or real-time event triggering strategy, thereby improving user stickiness and game revenue; the use of a distributed architecture combined with stream and batch processing can be flexibly expanded according to business needs, improving decision-making support capabilities, whether it is to increase the processing capacity of data streams or expand storage space or computing resources, it can be adjusted without affecting the stability of the overall system, ensuring the long-term and efficient operation of the system; integrating real-time analysis and historical data processing, not only It helps monitor the current gaming environment in real time and predict future trends based on historical data, thus enhancing the decision-making support capabilities. Through intelligent data mining, it can provide managers with more comprehensive insights to help them make more scientific decisions. Through accurate analysis of real-time data, it can better understand players' needs and behaviors, optimize the gaming experience, and make timely responses based on abnormal monitoring, which can significantly improve players' satisfaction and stickiness and enhance the long-term competitiveness of the game. However, the existing technology generally only focuses on one of the real-time data stream processing or historical data processing, lacks a comprehensive processing method that combines real-time and historical data, resulting in incomplete and detailed data processing, and the depth and breadth of data processing are not ideal.

[0017] In one embodiment of the present invention, the S22 includes: According to business needs and data traffic, evaluate the scale of the Kafka cluster, including the number of servers, hardware configuration, and network bandwidth; the purpose of the evaluation is to ensure that the cluster can meet the stable transmission and storage requirements of real-time data streams; according to the evaluation results, deploy the Kafka cluster and perform detailed configuration, including setting up the Zookeeper cluster to ensure the high availability of the Kafka cluster, configuring the Kafka broker node, and setting up necessary security measures (such as SSL / TLS encryption, identity authentication, etc.); Develop a reasonable partitioning strategy based on data characteristics and business needs; divide data into multiple partitions based on factors such as data type, size, and access frequency; and configure Kafka's replica strategy, including setting the number of replicas for each partition, and configuring the replica election and synchronization mechanism; Configure Kafka producers to receive real-time data streams from game servers; including setting parameters such as the producer's serialization method, compression algorithm, batch sending size, etc., and store the received real-time data streams in a distributed storage system (such as HDFS or HBase) through Kafka's consumer interface; during the storage process, factors such as data format, storage path, and index strategy need to be considered to ensure data readability and queryability; at the same time, set up necessary data backup and recovery mechanisms to ensure data persistence and security; Establish a monitoring and alarm mechanism for the Kafka cluster to monitor the performance indicators of the cluster (such as throughput, latency, error rate, etc.) in real time; if an anomaly or performance bottleneck is found, trigger an alarm immediately and take appropriate measures.

[0018] The working principle and effect of the above technical solution are as follows: by reasonably evaluating the scale, hardware configuration and network bandwidth of the Kafka cluster, it is ensured that the cluster can handle a large amount of real-time data streams and ensure the stability of the system, thereby improving the stability of the real-time data stream; in addition, configuring the Zookeeper cluster to provide high availability for Kafka can prevent the cluster from being interrupted due to node failures, ensure the reliable transmission and storage of data, and guarantee the quality of data; reasonable partitioning strategies and replica mechanisms can ensure load balancing and efficient reading of data; by designing partitions according to data characteristics, the storage structure and access performance of data can be optimized, the load balancing capability of data can be improved, and the The single point failure risk of the system; at the same time, the copy mechanism enhances the fault tolerance of the data, ensuring that the data can still be accessed normally when the cluster fails; by carefully configuring the Kafka producer, the real-time data stream generated by the game server can be efficiently transmitted to the Kafka cluster, improving the throughput, write reliability and low-latency transmission of data transmission; reducing the pressure of network bandwidth, the risk of data loss and system resource consumption; enhancing the scalability of the system, the real-time nature of data processing, fault tolerance and data consistency; the serialization method, compression algorithm and batch sending settings of the producer will directly affect the efficiency and stability of data transmission, thereby ensuring the real-time processing capability of large-scale data streams. In addition, by configuring the consumer interface, data can be reliably stored in a distributed storage system (such as HDFS or HBase), improving the security of the data; configuring Kafka's security measures (such as SSL / TLS encryption and authentication) can effectively ensure the security of data during transmission; the setting of the data backup and recovery mechanism further ensures the persistence and reliability of the data, prevents data loss and can quickly recover after a failure; establish a complete monitoring and alarm mechanism, track the performance indicators of the Kafka cluster in real time, and can promptly discover performance bottlenecks or abnormal situations. Through the automated alarm mechanism, the system can take appropriate measures immediately when problems occur to avoid affecting business continuity; by storing data in a distributed storage system, it can support the storage and query of massive data, improving the convenience of data acquisition; reasonable storage paths and indexing strategies can increase the speed of data retrieval and help business analysis and decision-making; the deployment of the Kafka cluster, the partitioning strategy and the design of the replica mechanism can be horizontally expanded according to business needs and the growth of data volume. When faced with an increase in data volume or business expansion, the system can respond flexibly and maintain efficient operation; by flexibly configuring Kafka producers and consumers, the compatibility of different data sources and storage systems can be ensured; whether it is real-time data stream processing or subsequent data analysis, it can adapt to different business needs and enhance overall business support capabilities.The technical solution of the present application greatly improves the efficiency and accuracy of data processing through Kafka cluster and partition processing when comprehensively processing historical data and real-time data. The traditional processing method mainly processes single data, resulting in a lack of ability to comprehensively process data and inability to achieve efficient processing. Due to the correlation between real-time data and historical data, the fault tolerance of data will be greatly increased when processing single data.

[0019] In one embodiment of the present invention, the S23 includes: S231. Deploy and configure the Apache Flink cluster according to business needs and data traffic, including determining the cluster size, node configuration, and task parallelism; connect Kafka as a data source to Flink and define real-time data streams, including configuring Kafka consumer parameters, setting the serialization method of data streams, and defining the schema of data streams; S232, preprocessing the real-time data stream, and extracting key features in the real-time data stream based on business requirements, such as player behavior features and game status features; S233. Use Flink's window operations, state management and other features to perform real-time analysis on real-time data streams, including calculating players' real-time behavior patterns, game popularity changes and other indicators, and performing real-time game anomaly detection; based on the extracted features and real-time analysis results, build an anomaly detection model based on machine learning algorithms; S234. According to business requirements, set alarm rules, including defining the threshold of abnormal behavior, triggering conditions of alarm, level of alarm, etc.; once abnormal behavior is detected, the alarm mechanism is triggered.

[0020] The working principle and effect of the above technical solution are as follows: by deploying and configuring the Apache Flink cluster, the cluster size can be dynamically adjusted according to business needs and data traffic to ensure the high efficiency of real-time data stream processing; Flink's stream processing capabilities and low latency characteristics can ensure real-time data transmission and processing, reduce data lag, and improve system response speed; Kafka is connected to Flink as a data source, which can seamlessly transmit high-throughput real-time data streams, improve real-time data processing capabilities, reduce system load imbalance, and enhance system scalability; by configuring Kafka consumer parameters, setting the serialization method of data streams, and defining the Schema of data streams, it can ensure that data is transmitted from K The transmission from afka to Flink is stable and reliable, and ensures the uniformity of data format and type, reducing the complexity and error rate of data parsing; pre-processing of real-time data streams based on business needs can accurately extract key features of the game process (such as player behavior, game status, etc.), provide accurate input for subsequent data analysis and model building, and quickly identify player behavior patterns and key changes in the game; Flink's window operation and state management functions support real-time computing and analysis, enabling instant analysis of real-time data streams and calculation of dynamic indicators such as player behavior patterns and game popularity, improving the accuracy and timeliness of real-time data analysis and the computing power of dynamic business indicators. And the ability to process large-scale data streams; reduce data processing delays, resource overhead caused by state management, and business indicator volatility problems; by introducing anomaly detection models established by machine learning algorithms, it can intelligently identify potential abnormal behaviors or patterns in the game, improve the system's responsiveness, and avoid the spread of abnormal situations in the game environment; by setting detailed alarm rules, it can flexibly configure alarm conditions according to different business needs, improve the flexibility of fault handling, set thresholds, trigger conditions, and alarm levels, so that when abnormal behavior is detected, it can trigger alarms in time to ensure that relevant personnel can respond and handle it as soon as possible, reducing business risks and losses caused by abnormal behavior; through Through real-time analysis, feature extraction and anomaly detection of real-time data streams, the system can monitor and evaluate the quality and integrity of data streams in real time to ensure the healthy operation of data streams. At the same time, it can promptly discover potential problems and take measures to enhance the stability of the system. By analyzing changes in player behavior and game status in real time, it can provide real-time data support for business decisions. For example, it can adjust game strategies or content according to real-time changes in popularity, or respond quickly to abnormal behaviors, thereby improving the flexibility and market competitiveness of game operations. The Flink cluster deployment and configuration in the solution can be elastically expanded according to the amount of data and business needs, ensuring that the system can cope with growing data traffic and complex business needs.In addition, the combination of Kafka and Flink supports various complex real-time data processing requirements, has high flexibility, and can adapt to changing business scenarios.

[0021] In one embodiment of the present invention, the S233 includes: Use Flink's time windows (such as TumblingWindow and SlidingWindow) to segment real-time data streams. Time windows are used to define the aggregation and analysis of data within a fixed time interval, while sliding windows allow data windows to slide on the time axis. Apply Flink’s state management mechanism to maintain a state for each key (such as player ID, game ID) in the real-time data stream. The state is used to store and process cumulative information across time windows, such as the cumulative number of player logins and the total amount of in-game spending. Based on the preprocessed real-time data stream, use Flink's DataStream API to implement complex business logic and mine players' real-time behavior patterns, including statistics on players' login frequency, game time, activity, etc., as well as analysis of players' behavior paths and preferences within the game. By analyzing various in-game indicators (such as the number of online users, game time, item purchases, etc.) in real time, a game popularity evaluation model is constructed. The game popularity evaluation model is used to dynamically reflect the popularity of the game and the player participation, and provide real-time feedback for game operations; Perform feature selection and feature engineering based on the extracted real-time data stream features (such as player behavior features, game status features); perform anomaly detection on real-time data streams based on machine learning algorithms; identify possible abnormal behaviors or events by calculating the anomaly score or probability of the data; The anomaly detection model is trained using normal samples in historical data and real-time data streams; at the same time, a model update mechanism is established to continuously update and optimize model parameters and structures based on newly emerging abnormal behaviors and patterns.

[0022] The working principle and effect of the above technical solution are as follows: through Flink's time windows (such as TumblingWindow and SlidingWindow), real-time data streams can be segmented and aggregated, which helps to efficiently calculate data within a fixed time interval, and flexibly process data according to different time windows, ensuring the efficiency and real-time nature of data analysis, improving the precision and accuracy of data calculation, reducing memory and storage overhead and computational complexity, and enhancing real-time monitoring and analysis capabilities; Flink's state management mechanism can maintain an independent state for each data key (such as player ID, game ID), enhancing the ability to accumulate data across time windows, and when processing player behavior and game status, it can accurately track and calculate long-term indicators such as cumulative login times and total in-game consumption, providing a reliable basis for subsequent analysis and decision-making; through Flink's DataStreamAPI, complex real-time business logic can be implemented. For example, it can accurately count players' login frequency, game time, and activity, and analyze players' behavior paths and preferences, providing game companies with rich insights into player behavior, helping to accurately adjust operational strategies and optimize user experience, and enhance user stickiness; by real-time analysis of various key indicators in the game (such as the number of online users, game time, item purchases, etc.), a dynamic game popularity evaluation model can be built to reflect the popularity of the game and player participation. This model provides instant feedback for game operations, helping the operation team to identify and respond to changes in game popularity in a timely manner, and ensure the long-term appeal of the game; by performing feature extraction and feature engineering on real-time data streams, abnormal patterns of player behavior can be effectively identified. The anomaly detection model based on machine learning algorithms can timely detect potential abnormal behaviors or events, such as malicious behaviors, cheating behaviors, or system failures, by calculating anomaly scores or probabilities, which helps to improve the stability of the game environment and ensure that the player experience is not affected. Based on normal samples in historical data and real-time data streams, the anomaly detection model can be trained, and through a continuous update mechanism, the model parameters and structure are optimized according to newly emerging abnormal behaviors and patterns, which can continuously optimize the anomaly detection mechanism and continuously improve system performance. Traditional anomaly detection is usually performed through a single static model, which makes it impossible to adapt to dynamic data, further resulting in insufficient accuracy and low efficiency of anomaly detection. By real-time monitoring of player behavior and game popularity and providing accurate operational data support, it can help game operation teams respond quickly, enhance user experience, and improve operational efficiency and business performance. The combination of Flink and Kafka provides a highly scalable architecture that can cope with the processing needs of large-scale data streams.As data volume and business demands continue to grow, the system can be flexibly expanded to ensure efficient real-time data processing; in addition, Flink's powerful functions support a variety of complex data processing requirements and have good flexibility.

[0023] In one embodiment of the present invention, S3 includes: S31. According to business needs and data characteristics, adopt feature selection strategies based on statistics, model correlation or business experience to screen the most representative features; and pre-process the features; S32. Use Cartesian product to generate new feature combinations and enrich the feature space; verify and optimize the newly generated features, and adjust the feature crossover strategy and method according to the verification results; S33. According to business requirements and data distribution, the preprocessed feature data is divided into a training set, a validation set, and a test set by using time series partitioning; S34. Manage and store the divided data sets; at the same time, set data access permissions and backup policies to prevent data leakage and loss.

[0024] The working principle and effect of the above technical solution are as follows: by selecting the most representative features according to business needs, data characteristics and industry experience, and performing preprocessing, the performance and efficiency of the model can be effectively improved; feature selection can remove irrelevant or redundant features, reduce data noise, and improve the accuracy and generalization ability of the model; preprocessing ensures data format standardization and consistency, and further improves the stability of model training; using Cartesian products to generate new feature combinations helps to enrich the feature space, provide more useful information for the model, and help the model better capture the complex relationships and interaction effects in the data, thereby improving the model's predictive ability; using a time series partitioning strategy to divide the data into training sets, validation sets, and test sets can ensure that the model's training and evaluation processes follow a chronological order and avoid future data leakage into the training process; this partitioning method is applicable For time series data analysis, it can more realistically reflect the performance of the model in actual business and ensure the fairness of training and testing; the security and availability of data are improved through data set management and storage strategies; by setting data access permissions, it can control the access to data by different roles and personnel to prevent data leakage or abuse; at the same time, the backup strategy can prevent data loss and ensure the persistence and reliability of data; good data management enhances data governance and compliance; by adjusting the feature cross-strategy and method according to the verification results, the feature engineering process can be made more flexible and adaptable to changing business needs, and the effect of feature selection and combination can be improved, so that model training is always kept in the optimal state; reasonable data partitioning and diversified feature combinations help to improve the adaptability and generalization ability of the model, especially in the face of complex business scenarios and changes in data distribution.

[0025] In one embodiment of the present invention, the S4 includes: S41. Select a federated learning framework (such as TensorFlowFederated (TFF)) and build a federated learning platform; and configure the parameters and strategies of the federated learning platform, including communication protocols, model aggregation algorithms, data security, and privacy protection strategies; S42. Assign training tasks to each participant (such as game server, player device, etc.) based on business needs and data distribution. Each participant uses local data to train a local model and uploads the model parameters to the federated learning platform. At the same time, the training process is monitored and optimized in real time through training logs and monitoring mechanisms. S43. The federated learning platform aggregates the local models of each participant to generate a global model; uses the validation set to validate the global model and evaluate the performance of the model; adjusts model parameters, optimizes feature selection, or improves training strategies based on the validation results; S44. Deploy the trained global model to the production environment to provide real-time prediction and classification services for game operations. Based on the model monitoring mechanism, monitor the performance, stability and accuracy of the model in real time. At the same time, tune and update the model according to the monitoring results.

[0026] The working principle and effect of the above technical solution are as follows: by adopting a federated learning framework, data is ensured to always remain on the local device or server without being transmitted to a central server, thereby improving data privacy and security; it helps to ensure that sensitive data (such as user behavior data, personal information, etc.) will not be leaked, thereby improving the data safety line; training tasks are assigned to various participants (such as game servers, player devices, etc.) to achieve distributed training, which can make full use of the computing resources of all parties, improve training efficiency, and improve the utilization efficiency of computing resources, training speed, and model performance; each participant only needs to train a local model and upload model parameters instead of transmitting a large amount of raw data, reducing communication costs and time delays as well as the load on centralized computing resources; since each participant uses local data for training, a personalized local model can be generated; after model aggregation, the generated global model can take into account the characteristics of each participant, thereby achieving more accurate prediction and classification services, enhancing the generalization ability of the global model, speeding up the training process, and improving training efficiency. For example, in game operations, accurate recommendations or predictions can be provided based on multi-dimensional data such as player behavior characteristics and device status; by using the real-time monitoring and tuning mechanism of the federated learning platform, the training process can be dynamically monitored, and model parameters can be adjusted, feature selection can be optimized, and training strategies can be improved based on the verification results. In this way, the model can adapt to changing business needs and data distribution to ensure its long-term effectiveness and accuracy. By real-time monitoring of the training process, potential problems can be discovered and tuned in a timely manner. For example, adjustments can be made immediately when model performance degradation or stability problems are discovered. The federated learning platform ensures that the model always maintains high accuracy and stability through model update and tuning strategies; the architecture of federated learning avoids large-scale centralized storage and transmission of data, and only needs to transmit the parameters of local models of all parties, thereby reducing the consumption of network bandwidth and the pressure of data storage and transmission, which is especially suitable for scenarios with large amounts of data (such as games, large platforms, etc.); after training and verification, the global model can be deployed to the production environment in a timely manner to provide real-time services for actual business. In addition, based on the model monitoring mechanism, the performance, stability and accuracy of the model can be continuously monitored, and automatic tuning and updating can be performed based on the monitoring results, ensuring the stability and long-term effectiveness of the model in practical applications; the federated learning platform promotes collaboration among multiple participants and avoids the phenomenon of data silos. Without sharing data, multiple parties can share the results of model improvements through joint training and aggregation of models. This helps to improve the generalization ability and application effect of the model, especially when training on different devices and platforms, to better meet the needs of all parties.

[0027] In one embodiment of the present invention, the S42 includes: Screen out suitable participants, such as game servers and player devices, based on business needs and data distribution characteristics; screening criteria include data quality, data scale, computing resources, and network conditions; conduct resource evaluation on the screened participants, including computing resources (such as CPU, GPU, memory, etc.), storage resources, network bandwidth, etc.; assign training tasks based on the evaluation results; Decompose the overall training task into multiple subtasks, each subtask corresponds to a participant, and each subtask is independent; According to the characteristics and resource conditions of the participants, formulate appropriate training strategies, including selecting appropriate training algorithms, setting reasonable training rounds, and defining the model aggregation cycle; configure a local training environment for each participant, including installing necessary software (such as TensorFlowFederated), configuring model parameters, preparing training data, etc.

[0028] Each participant uses local data to train a local model in a local environment based on a distributed training algorithm. After the training is completed, each participant uploads the parameters of the local model to the federated learning platform through an encryption algorithm. During the training process, key indicators are recorded, including training logs, including training time, training rounds, loss function value and accuracy. Based on the real-time monitoring mechanism, the training process is monitored in real time. If an abnormal situation is found (such as a decrease in training speed or accuracy), an alarm is triggered and corresponding measures are taken to optimize or adjust the training strategy.

[0029] The working principle and effect of the above technical solution are as follows: by reasonably screening participants and conducting resource evaluation according to the characteristics of the participants (such as data quality, computing resources and network conditions), the training tasks and the resources of the participants can be effectively matched, thereby optimizing the task allocation, improving the computing efficiency, avoiding excessive allocation of resources to devices with weaker computing power, and thus saving resource overhead; decomposing the overall training task into multiple subtasks and assigning them to each participant for training respectively, which can realize true distributed training, avoid excessive dependence on the central server, and improve the reasonable utilization of resources; each participant conducts training independently locally, reducing the data transmission. The system reduces the burden of training, and through local computing, it reduces the pressure on network bandwidth and improves the efficiency of the entire training process. According to the characteristics and resource conditions of the participants, it customizes appropriate training strategies (such as selecting training algorithms, setting training rounds, and defining model aggregation cycles) to ensure that each participant can train efficiently under conditions that suit them, thereby improving the overall training effect. Differentiated strategies for different devices or platforms enable each subtask to achieve maximum training performance. By training the model locally and uploading the model parameters, and then aggregating to form a global model, it can fully consider the data characteristics of each participant, so that the trained global model can better adapt to diverse data. Distribution improves the generalization ability and accuracy of the model; the local training model only transmits parameters instead of raw data, thereby ensuring data privacy and avoiding the risk of data leakage; at the same time, the use of encryption algorithms to transmit model parameters further improves the security of the system and ensures that the data of the participants is not abused; by real-time monitoring of the training process and recording key indicators (such as training time, loss function value, accuracy, etc.), problems in training can be discovered and solved in a timely manner; when training is abnormal (such as a decrease in training speed or accuracy), the system will trigger an alarm and can automatically adjust the training strategy or perform tuning to ensure the stability and quality of training; through distributed training The local model parameters are uploaded and encrypted, which avoids large amounts of data transmission and reduces the storage requirements for the central server. Especially in scenarios with large amounts of data, it reduces the consumption of communication bandwidth and improves the overall response speed and processing capacity of the system. The solution can support multiple different types of devices or platforms to participate in training. Without relying on a powerful central server, it can adapt to the growing number of participants and data scale through reasonable task allocation and optimization strategies, and has strong scalability and flexibility. Under multi-party collaboration, each participant can share the progress of the model without having to exchange data directly, maintaining the isolation and security of the data, while improving the collaboration efficiency of all parties. Through model aggregation, participants can gain experience and feedback from the training of other parties to improve their own models.

[0030] In one embodiment of the present invention, the S43 includes: Before aggregation, the local model parameters uploaded by each participant are preprocessed, and the local model parameters of each participant are weighted averaged according to the selected aggregation algorithm (such as federated average, weighted average, etc.) to generate global model parameters; Verify the aggregated global model parameters. At the same time, identify and handle possible aggregation anomalies (such as data anomalies, model parameter anomalies, etc.) through the exception handling mechanism. Prepare an independent validation set to evaluate the performance of the global model; set evaluation metrics, such as accuracy, recall, F1 score, etc., to comprehensively measure the performance of the model; use the validation set to validate the global model, calculate the evaluation metrics, and comprehensively evaluate the performance of the model; Analyze the validation results to identify the reasons for poor model performance, such as improper feature selection, unreasonable model structure, etc.; at the same time, locate performance bottlenecks; According to the verification results and problem location, formulate parameter adjustment and optimization strategies, including adjusting hyperparameters such as learning rate, batch size, regularization parameters, optimizing feature selection, improving model structure, etc.; apply the adjusted parameters and optimization strategies to the global model and perform iterative training; gradually optimize the performance of the global model through continuous iteration; at the same time, record the results of each iteration; Set the convergence judgment criteria and stop conditions; when the performance of the global model reaches the preset convergence criteria or stop conditions, stop the iterative training and save the optimized global model as the final model.

[0031] The working principle and effect of the above technical solution are as follows: by weighted averaging the local model parameters and combining the training results of different participants, a more accurate global model can be generated; the weighted average algorithm can be adjusted according to the training quality of each participant, thereby improving the stability and generalization ability of the global model; the local model parameters are preprocessed before aggregation, and potential problems such as data anomalies or model parameter anomalies are identified and handled through the exception handling mechanism, effectively avoiding the degradation of global model performance due to problems of a single participant, thereby ensuring the stability and reliability of the aggregation process; by preparing an independent validation set and setting multi-dimensional evaluation indicators (such as accuracy, recall rate, F1 score, etc.), the performance of the global model can be comprehensively evaluated from multiple angles. This helps to discover the advantages and disadvantages of the model and provide quantitative evaluation standards; after evaluating the model performance, the results are analyzed and the problems are located (such as improper feature selection, unreasonable model structure, etc.), providing directions for improvement of the model. By formulating parameter adjustment and optimization strategies (such as adjusting hyperparameters such as learning rate and batch size), the model can be gradually optimized, performance bottlenecks can be resolved, and the model effect can be improved; through iterative training and optimization, the solution can automatically tune the model, continuously improve the global model, and ensure the improvement of model performance; recording the results of each iteration also helps to track the optimization process and provide a reference for subsequent adjustments; during the optimization process, adjusting regularization parameters and improving feature selection can effectively prevent the model from overfitting problems and improve the model's generalization ability on unknown data, thereby enhancing the model's practical application value; by setting convergence judgment criteria and stopping conditions, it can be The training process is terminated at an appropriate time to avoid unnecessary calculations and waste of resources. When the model performance reaches the preset standard, the iteration can be stopped to ensure that the training process is efficient and non-redundant. Through the above-mentioned multi-level optimization and iterative training, the system can reduce excessive computing resource consumption and improve training efficiency while ensuring accuracy. At the same time, adjustments and optimizations for each iteration make the system more flexible and adaptable to different data and needs. During each iteration and optimization process, corresponding adjustments can be made according to the changes in the current data and the performance of the model to ensure that the global model can continue to adapt to new data distribution and needs, thereby maintaining high performance during long-term operation.

[0032] According to one embodiment of the present invention, a data processing system based on a gaming platform comprises a memory, a processor and a computer program stored in and executable on the memory, wherein the processor executes the program to implement any of the data processing methods based on the gaming platform described above.

[0033] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A data processing method based on a game platform, characterized in that: The method comprises: S1. Collect multi-source data; S2. Build a real-time data stream processing environment; use Kafka to receive real-time data streams and store them in a distributed file system or columnar database; use the stream processing engine to perform instant analysis on real-time data streams and extract valuable information; use batch processing tools to batch process and analyze historical data stored in HDFS or HBase to mine potential data patterns and trends; S3, perform feature engineering processing; S4. Aggregate the local models of each participant through the federated learning platform.

2. The data processing method based on the game platform according to claim 1, characterized in that: Said S1 comprises: S11. Identify multiple types of data sources, perform preliminary cleaning and integration of data sources, and form a unified data warehouse; S12, processing structured data and unstructured data; S13, extract key features; S14. Filter the most representative features; build a three-dimensional model of the game world; S15. Build a digital twin model of the game world and synchronize virtual and real world.

3. The data processing method based on the game platform according to claim 1, characterized in that: The S2 comprises: S21. Select a stream processing framework based on business needs and build a real-time data stream processing environment; S22. Configure the Kafka cluster and set the partition and replica strategies; use Kafka to receive real-time data streams and store them in a distributed storage system; S23. Use the stream processing engine to analyze the real-time data stream in real time. At the same time, set up an alarm mechanism to monitor and warn abnormal behaviors in real time. S24. Load historical data from the distributed storage system; use batch processing tools to batch process and analyze historical data to explore potential data patterns and trends; at the same time, combine machine learning algorithms to predict and classify historical data.

4. The data processing method based on the game platform according to claim 3 is characterized in that: The S22 comprises: Evaluate the scale of the Kafka cluster based on business needs and data traffic, deploy the Kafka cluster based on the evaluation results, and perform detailed configuration; Develop a reasonable partitioning strategy based on data characteristics and business needs; divide data into multiple partitions; and configure Kafka's replica strategy; Configure the Kafka producer to receive the real-time data stream from the game server; store the received real-time data stream in the distributed storage system through the Kafka consumer interface; Establish a monitoring and alarm mechanism for the Kafka cluster to monitor the performance indicators of the cluster in real time; if an anomaly or performance bottleneck is found, trigger an alarm immediately and take appropriate measures.

5. The data processing method based on the game platform according to claim 3 is characterized in that: The S23 comprises: S231. Deploy and configure the Apache Flink cluster according to business requirements and data traffic, connect Kafka as a data source to Flink, and define real-time data streams; S232, preprocessing the real-time data stream, and extracting key features in the real-time data stream based on business requirements; S233. Use Flink's window operations and state management features to perform real-time analysis on real-time data streams; build an anomaly detection model based on machine learning algorithms based on the extracted features and real-time analysis results; S234. Set alarm rules according to business needs, and trigger the alarm mechanism once abnormal behavior is detected.

6. The data processing method based on the game platform according to claim 5, characterized in that: The S233 includes: Use Flink's time window to segment the real-time data stream; Apply Flink’s state management mechanism to maintain a state for each key in the real-time data stream; Based on the preprocessed real-time data stream, use Flink's DataStream API to mine players' real-time behavior patterns and analyze their behavior paths and preferences within the game. Build a game popularity evaluation model by analyzing various in-game indicators in real time; Perform feature selection and feature engineering based on the extracted real-time data stream features; perform anomaly detection on real-time data streams based on machine learning algorithms; identify possible abnormal behaviors or events by calculating the anomaly score or probability of the data; The anomaly detection model is trained using normal samples in historical data and real-time data streams; at the same time, a model update mechanism is established to continuously update and optimize model parameters and structures based on newly emerging abnormal behaviors and patterns.

7. The data processing method based on the game platform according to claim 1, characterized in that: The S3 includes: S31, screening the most representative features and preprocessing the features; S32, generate new feature combinations to enrich the feature space; S33, dividing the preprocessed feature data into a training set, a validation set and a test set; S34. Manage and store the divided data sets.

8. The data processing method based on the game platform according to claim 1, characterized in that: The S4 comprises: S41. Build a federated learning platform and configure the parameters and strategies of the federated learning platform; S42, assigning training tasks to each participant, and each participant uses local data to train a local model; S43. The federated learning platform aggregates the local models of each participant to generate a global model; S44. Deploy the trained global model to the production environment.

9. The data processing method based on the game platform according to claim 8, characterized in that: The S42 includes: Screen participants, conduct resource assessment on the selected participants, and assign training tasks based on the assessment results; Decompose the overall training task into multiple subtasks, each subtask corresponds to a participant, and each subtask is independent; Develop appropriate training strategies based on the characteristics and resource conditions of the participants; Each participant uses local data to train a local model in a local environment based on a distributed training algorithm. After the training is completed, each participant uploads the parameters of the local model to the federated learning platform through an encryption algorithm. During the training process, key indicators are recorded and the training process is monitored in real time based on the real-time monitoring mechanism. If any abnormal situation is found, an alarm is triggered and corresponding measures are taken to optimize or adjust the training strategy.

10. A data processing system based on a game platform, characterized in that: It comprises a memory, a processor and a computer program stored in the memory and executable on the memory, wherein the processor executes the program to implement a data processing method based on a gaming platform as described in any one of claims 1 to 9.

Images