Data processing method and apparatus, computer program product, and electronic device

Abstract

A data processing method and device, a computer program product and an electronic device, relate to the technical field of computers. The method comprises: obtaining target data based on a data acquisition request, converting the target data into intermediate state data, filtering the target data based on the intermediate state data to obtain first data; analyzing the first data to obtain a virtual object set included in the first data, aggregating virtual objects in the first data according to the virtual object set to obtain a target virtual object set; extracting a data chain node associated with a virtual object in the virtual object set and second data corresponding to the data chain node from the first data, and storing the data chain node and the second data. The entity relationship chain of the virtual object in the target data is retained, and the data processing efficiency is improved.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a data processing method and apparatus, a computer program product, and an electronic device. Background Technology

[0002] Massive multiplayer online games (MMORPGs) require real-time monitoring of the status of tens of thousands of virtual objects. One relevant technology is a game server monitoring solution based on Redis and Flink, whose core process includes:

[0003] During game runtime, the process provides information such as the load of the region and virtual objects to the microservices. The microservices then use `json.dump` to store the collected information in Redis in the format of key:serverGroup, value:zlib+base64. Flink pulls all data (from all clusters requiring data collection) from the microservice cluster every 65 seconds via HTTP (queried from MongoDB). The JSON data is then parsed by `GameInfoParser`, categorized by ServerGroup using `KeyBy`, expired data is filtered out using a time filter, and finally aggregated into various dimensional data tables using `StaticInfoToPerfData` for different query conditions.

[0004] However, the above method has two problems. First, it performs data aggregation too early in the Flink processing stage, which destroys the multi-dimensional relationship information in the original data. When a new query dimension is needed, the Flink job must be modified and the aggregation table must be rebuilt. Second, in order to meet different query requirements, multiple aggregation tables need to be maintained. When the cluster size increases, the management cost increases dramatically. The complex aggregation logic exists in Flink in a hard-coded form. When it is necessary to backtrack historical data, the original data has been aggregated and the complete entity relationship chain cannot be reconstructed.

[0005] Based on the above problems, a new data processing method is needed. Summary of the Invention

[0006] This disclosure provides a data processing method to at least partially solve the problem of low data processing efficiency in related technologies.

[0007] According to a first aspect of this disclosure, a data processing method is provided, the method comprising:

[0008] Based on the data acquisition request, target data is acquired, the target data is converted into intermediate data, and the target data is filtered based on the intermediate data to obtain first data;

[0009] The first data is parsed to obtain a set of virtual objects included in the first data. The virtual objects in the first data are aggregated according to the set of virtual objects to obtain a target set of virtual objects.

[0010] Extract the data chain node associated with the virtual object in the virtual object set and the second data corresponding to the data chain node from the first data, and store the data chain node and the second data.

[0011] According to a second aspect of this disclosure, a data processing apparatus is provided, the apparatus comprising:

[0012] The target data acquisition module is used to acquire target data based on a data acquisition request, convert the target data into intermediate data, and filter the target data based on the intermediate data to obtain first data;

[0013] The virtual object aggregation module is used to parse the first data to obtain a set of virtual objects included in the first data, and to aggregate the virtual objects in the first data according to the set of virtual objects to obtain a target set of virtual objects;

[0014] The data storage module is used to extract data chain nodes associated with virtual objects in the virtual object set and second data corresponding to the data chain nodes from the first data, and to store the data chain nodes and the second data.

[0015] According to a third aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the method of the first aspect described above and possible implementations thereof.

[0016] According to a fourth aspect of this disclosure, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to perform the method of the first aspect and possible implementations thereof by executing the executable instructions.

[0017] This disclosure provides a data processing method that, based on a data acquisition request, acquires target data, converts the target data into intermediate data, filters the target data based on the intermediate data to obtain first data, parses the first data to obtain a set of virtual objects included in the first data, aggregates the virtual objects in the first data according to the set of virtual objects to obtain a target set of virtual objects, and extracts data chain nodes associated with virtual objects in the set of virtual objects and second data corresponding to the data chain nodes from the first data, and stores the data chain nodes and the second data. On one hand, after acquiring the target data, converting it into intermediate data and deleting expired data from the intermediate data reduces memory usage and server resource consumption. On the other hand, after deleting expired data, parsing the first data to obtain the set of virtual objects included in the first data and aggregating the virtual objects according to the set of virtual objects simply aggregates different types of virtual objects into a table. For each virtual object in the target set of virtual objects, determining the data chain node corresponding to each virtual object and the second data corresponding to the data chain node in the first data, and storing the data chain node and the second data corresponding to the data chain node... The data is stored by extracting data chain nodes and storing the data chain nodes and the corresponding second data. The target data corresponding to the virtual object is not aggregated, thus preserving the data chain associated with the virtual object in the target data and avoiding the destruction of multidimensional relationships within the target data. This solves the problem of premature data aggregation in related technologies, which leads to the destruction of multidimensional relationships in the original target data. It simplifies Flink's data processing logic and code, reducing system complexity. Furthermore, when there are new data display requests, no modification to the Flink code is required; only the SQL statement executed by the query backend needs to be modified, improving data processing efficiency. Attached Figure Description

[0018] Figure 1 A flowchart illustrating a data processing method in this exemplary embodiment is shown;

[0019] Figure 2 This exemplary embodiment shows a flowchart of a method for obtaining a target number based on the data acquisition request.

[0020] Figure 3 A schematic diagram illustrating a data acquisition process in this exemplary embodiment is shown;

[0021] Figure 4 This exemplary embodiment shows a flowchart of a method for converting the target data into intermediate data.

[0022] Figure 5 This example embodiment shows a flowchart of a method for filtering target data based on intermediate state data to obtain first data.

[0023] Figure 6 This example embodiment illustrates a method flowchart for parsing the first data to obtain a set of virtual objects included in the first data, and aggregating the virtual objects in the first data according to the set of virtual objects to obtain a target set of virtual objects.

[0024] Figure 7 This example embodiment shows a flowchart of a method for extracting a data chain node associated with a virtual object in the virtual object set from the first data, the second data corresponding to the data chain node, and storing the data and the second data.

[0025] Figure 8 A schematic diagram of a data processing apparatus according to this exemplary embodiment is shown;

[0026] Figure 9 A schematic diagram of the structure of an electronic device in this exemplary embodiment is shown. Detailed Implementation

[0027] Exemplary embodiments of this disclosure will be described more fully below with reference to the accompanying drawings.

[0028] The accompanying drawings are schematic illustrations of this disclosure and are not necessarily drawn to scale. Some block diagrams shown in the drawings may be functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in hardware modules or integrated circuits, or in networks, processors, or microcontrollers. Implementations can be carried out in various forms and should not be construed as limited to the examples set forth herein. The features, structures, or characteristics described in this disclosure can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a thorough description of embodiments of this disclosure. However, those skilled in the art will recognize that one or more specific details may be omitted when implementing the technical solutions of this disclosure, or other methods, components, apparatuses, steps, etc., may be used to replace one or more specific details.

[0029] Massive multiplayer online games (MMORPGs) require real-time monitoring of the status of tens of thousands of virtual objects. Traditional game operations and maintenance rely on offline analysis of log files, resulting in latency of up to hours. To address this issue, a game server monitoring solution based on Redis and Flink is adopted, with the core process as follows:

[0030] During game runtime, processes provide microservices with information such as the load of the regions they handle and virtual objects. The microservices then store all received process information in Redis (Remote Dictionary Server) using `json.dump` with the key: serverGroup, value: zlib+base64. Flink pulls all data (from all clusters requiring data collection) from the microservice cluster every 65 seconds via HTTP (queried from MongoDB). The JSON data is then parsed using `GameInfoParser`, categorized by `ServerGroup` using `KeyBy`, expired data is filtered out using a time filter, and finally aggregated into various tables using `StaticInfoToPerfData` for different query conditions. The data processed by Flink is ultimately stored in multiple tables within ClickHouse.

[0031] The above process presents several problems: 1. Data dimension solidification leads to loss of related information. Premature data aggregation during the Flink processing stage (stream processing stage) destroys multi-dimensional relationships in the original data. For example, it's impossible to query the combination of logicserver and IP simultaneously (because the relationship between these two fields wasn't preserved during aggregation). When adding new query dimensions (such as zoneIndex + machineIp), the Flink job must be modified and the aggregation table rebuilt. 2. System scalability and maintainability issues: Table structure bloat: To meet different query needs, multiple aggregation tables (such as EntityInfo, GameEntitySnapshot, ProcessLoad, ServerLineEntitySnapshot, TotalSnapshot, ZoneBoundSnapshot, etc.) need to be maintained, leading to a significant increase in management costs as the cluster size expands. 3. Difficult code maintenance: Complex aggregation logic exists in the Flink job in hard-coded form. 4. Difficult data completion: When historical data needs to be traced back, the original data has already been aggregated, making it impossible to reconstruct the complete entity relationship chain. 5. Tightly coupled architecture design: Data acquisition, processing, and storage stages adopt a strongly dependent, sequential pattern. Question 3: Performance bottleneck in large data volume scenarios, single transmission exceeding the limit: When the data volume of a single serverGroup exceeds 20MB, the HTTP request failure rate increases significantly.

[0032] In view of the above problems, exemplary embodiments of this disclosure provide a data processing method.

[0033] In this article, it is important to understand that the terms used are explained as follows:

[0034] Flink: Apache Flink is an open-source distributed stream processing framework that provides low-latency, high-throughput data processing capabilities. Its core features include exactly-once state consistency guarantees, event-time processing semantics, and a unified batch and stream computation model, making it suitable for scenarios such as real-time data analysis and event-driven applications.

[0035] API Source: refers to the interface or method used to define the data input source, mainly used in streaming data processing scenarios; the Source operator is the input end of the Flink processing flow, responsible for reading data from external systems (such as files, databases, networks, etc.) and converting it into a data stream for subsequent operations.

[0036] Clickhouse is an open-source columnar database management system for Online Analytical Processing (OLAP), renowned for its high-performance real-time queries. Its design utilizes columnar storage, a vectorization engine, and data compression technology to achieve sub-second response times, supporting aggregate analysis of large-scale datasets and finding wide application in log analysis, user behavior analysis, and other fields.

[0037] refer to Figure 1 As shown, the data processing method may include the following steps:

[0038] Step S110. Based on the data acquisition request, acquire the target data, convert the target data into intermediate data, filter the target data based on the intermediate data, and obtain the first data;

[0039] Step S120. Parse the first data to obtain the set of virtual objects included in the first data, and aggregate the virtual objects in the first data according to the set of virtual objects to obtain the target set of virtual objects;

[0040] Step S130. Extract the data chain node associated with the virtual object in the virtual object set and the second data corresponding to the data chain node from the first data, and store the data chain node and the second data.

[0041] In the above data processing method, based on a data acquisition request, target data is acquired, and first data included in the target data is acquired. Based on the data acquisition request, the target data is acquired, and the target data is converted into intermediate data. The target data is filtered based on the intermediate data to obtain the first data. The first data is parsed to obtain a set of virtual objects included in the first data. The virtual objects in the first data are aggregated according to the set of virtual objects to obtain a target set of virtual objects. Data chain nodes associated with virtual objects in the set of virtual objects and second data corresponding to the data chain nodes are extracted from the first data, and the data chain nodes and the second data are stored. On the one hand, after acquiring the target data, it is converted into intermediate data, and expired data in this intermediate data is deleted, reducing memory usage and server resource consumption. On the other hand, after deleting expired data, the first data is parsed to obtain the set of virtual objects included in the first data. The virtual objects are then aggregated based on this set, simply grouping different types of virtual objects into a single table. For each virtual object in the target set, the data chain node corresponding to each virtual object and the second data corresponding to that data chain node are determined in the first data. The data chain node and the data chain node are then... The corresponding second data is stored. This involves extracting data chain nodes and storing the data chain nodes and their corresponding second data. The target data corresponding to the virtual object is not aggregated, thus preserving the data chain associated with the virtual object in the target data. This avoids destroying the multidimensional association information in the target data, solving the problem of premature data aggregation in related technologies that leads to the destruction of multidimensional association information in the original target data. This simplifies Flink's data processing logic and code, reducing system complexity. Furthermore, when there is a new data display request, no modification to the Flink code is required; only the SQL (Structured Query Language) statement executed by the query backend needs to be modified, improving data processing efficiency.

[0042] The following will provide further explanation and description of steps S110-S130.

[0043] In step S110, target data is acquired based on the data acquisition request, the target data is converted into intermediate data, and the target data is filtered based on the intermediate data to obtain first data.

[0044] In Flink, a timestamp is generated every 65 seconds. The API Source in Flink is the interface used to define the source of data input, serving as the starting point for data input within the framework. The API Source sends an HTTP request to the microservice, carrying the timestamp. The microservice retrieves the target data from Redis and sends it back to the API Source. The target data in Redis is server cluster information, including process information reported to the server cluster, basic information about the server cluster, basic process information including the process's basic information and the region information of the region it runs on, and basic region information including the region's basic information and the collection of virtual objects within that region.

[0045] Due to the surge in data volume, the original 1-3MB of data may increase to 20-30MB. Testing revealed that when the data volume reached 20MB, data retrieval would time out or even fail, leading to unpredictable problems such as microservice crashes and data loss. Therefore, when retrieving data from Redis, microservices can use a paginated retrieval method. The microservice compresses and segments the retrieved data using zlib and base64 based on the current Redis data, returning the first page and the total number of pages to the API Source. The API Source then uses the total number of pages and the first page's data to retrieve subsequent pages in parallel, concatenates them, and processes them. Once the API Source receives the concatenated data, it decompresses it to obtain the target data. That is, the target data is obtained after decompressing the process reporting information. After obtaining the target data, preprocessing and time filtering can be performed on it.

[0046] In one exemplary embodiment, reference is made to Figure 2 As shown, obtaining the target data based on the data acquisition request includes:

[0047] Step S210. Send a data retrieval request to the microservice cluster, retrieve and save the first page of data sent by the microservice cluster and the total number of pages;

[0048] Step S220. Send the remaining data retrieval request to the microservice cluster in parallel, and receive the remaining page data sent by the microservice cluster;

[0049] Step S230. Concatenate the first page data with the remaining page data to obtain concatenated data, and decompress the concatenated data to obtain the target data.

[0050] The following will provide further explanation and description of steps S210-S230. For details, please refer to... Figure 3The data retrieval process is illustrated in the diagram. The API Source sends a data retrieval request to the microservice, carrying a timestamp. The microservice retrieves data from Redis based on this request. After retrieving the data, it compresses and shards the data according to the preset maximum page size. The first page of data and the total number of pages are sent back to the API Source, and the paginated data is stored in Redis. The key name format is page:{request_ts}:{serverGroup}:{pageNum}, where page is the first page of data, request_ts is the timestamp carried in the data retrieval request, serverGroup is the server cluster to which the compressed data belongs, and pageNum is the total number of pages. After the API Source obtains the first page of data and the total number of pages, it saves the first page of data. Then, it sends multiple data retrieval requests in parallel to the microservice based on the remaining page count. The number of these requests is equal to the number of remaining pages. The microservice retrieves the remaining page data from Redis and sends it to the API Source. The API Source then concatenates the received first page data and the remaining page data to obtain concatenated data. After obtaining the concatenated data, it decompresses it to obtain the target data. The preset maximum page size can be 4MB, but this example embodiment does not impose a specific limitation on it.

[0051] By using paginated fetching, the amount of data retrieved in a single data retrieval request is reduced, improving network transmission success rate and resolving the issue of large data packet transmission failures. Compression and fragmentation of the retrieved data reduce data transmission volume, lowering the transmission latency from microservices to Flink, avoiding HTTP request timeouts, and parallel data retrieval fully utilizes network bandwidth, significantly reducing data transmission time.

[0052] In an exemplary embodiment, after decompressing the compressed data, the obtained target data includes at least server cluster information, process information included in the server cluster information, region information included in the process information, and a set of virtual objects included in the region information. Specifically, the server cluster information may include basic information of the server cluster and process information. The basic information of the server cluster may include the identifier of the server cluster. The process information may include basic information of the process and region information. The basic information of the process may include the process identifier, the number of regions carried by the process, the server cluster to which the process belongs, the data reporting timestamp, the IP address of the machine, the process load, etc. The region information may include basic information of the region and a set of virtual objects. The basic information of the region may include the region number, region load, region boundary, and the type and quantity of virtual objects included in the region. Each set of virtual objects stores virtual objects of the same type. In this example embodiment, since there are many virtual objects in a multiplayer online open-world game, in order to facilitate the reporting of data for each virtual object by the thread, the game scene can be divided into multiple regions according to preset boundary coordinates. Each region may include one or more virtual objects, which may be game characters, non-player characters, monsters, buildings, etc. The virtual objects are not specifically limited in this example embodiment.

[0053] After receiving the target data, Flink's API Source sends the target data to Flink's StatisticPreTimeFilterParser operator, where preprocessing and filtering of the target data are performed. In multiplayer online games, the target data is continuously generated as an unbounded stream. To improve the speed and efficiency of data processing, the StatisticPreTimeFilterParser operator needs to perform state transformation on the target data after receiving it. That is, it converts each piece of data in the target data into intermediate state data, which is key-value pair structured data. After the transformation is completed, expired data can be determined based on the transformed structured data.

[0054] In one exemplary embodiment, reference is made to Figure 4 As shown, the process of converting the target data into intermediate data includes:

[0055] Step S410. For the first target data in the target data, obtain the server cluster information in the first target data, and traverse the server cluster information to obtain the process information included in the server cluster information;

[0056] Step S420. Obtain the first attribute data from the process information, use the first attribute data as the key and the first target data as the value, store the key-value pair to obtain the intermediate state data.

[0057] The following will further explain and illustrate steps S410 and S420. Specifically, after the StatisticPreTimeFilterParser operator receives the target data, in order to improve the efficiency of subsequent data processing, the first data can be stored as intermediate data. During storage, for each piece of first target data in the target data, the server cluster information included therein can be obtained. The server cluster information is parsed to obtain the process information included in the server cluster information. The process information includes multiple attributes. The first attribute data in the process information can be determined as the key of the intermediate data, and the first target data is used as the value. The key-value pair is stored to obtain the intermediate data. The number of first attribute data can be one or more. In this example embodiment, the number of first attribute data is not specifically limited.

[0058] In one exemplary embodiment, the first attribute data in the process includes at least the process identifier, the server cluster to which the process belongs, the timestamp of the data reported by the process, and the batch of data reported by the process. That is, the process identifier, the server cluster to which the process belongs, and the timestamp of the data reported by the process are extracted from the process information included in the first target data, and the extracted attribute data is used as the first attribute data.

[0059] In one exemplary embodiment, the first attribute data in the process may further include the process identifier, the server cluster to which the process belongs, the timestamp of the data reported by the process, and the batch of data reported by the process. In this example embodiment, the first attribute data is not specifically limited.

[0060] For example, after the StatisticPreTimeFilterParser operator receives the target data, for each piece of first target data in the target data, it can obtain the process identifier, the server cluster to which the process belongs, the timestamp of the data reported by the process, and the batch of data reported by the process. The obtained process identifier, the server cluster to which the process belongs, the timestamp of the data reported by the process, and the batch of data reported by the process are used as keys, and the first target data is used as values. The key-value pair is stored to obtain intermediate state data.

[0061] In one exemplary embodiment, after receiving the target data, the StatisticPreTimeFilterParser operator first needs to determine whether there is expired data in the target data. If expired data exists, it is deleted from the target data. For each received first target data piece, the first attribute data of the first target data is extracted, and the first attribute data is used as the key, with each first target data piece as the value, for storage to obtain intermediate state data. In the intermediate state data, it is only necessary to determine whether the first target data piece is expired based on the key.

[0062] In one exemplary embodiment, reference is made to Figure 5 As shown, the step of filtering the target data based on the intermediate state data to obtain the first data includes:

[0063] Step S510. Using the key differentiation operator, classify the intermediate state data according to the process identifier in the first attribute data and the server cluster to which the process belongs, to obtain classified data;

[0064] Step S520. Based on the timestamp of the reported data of the process, determine the expired data included in the classified data, delete the expired data in the target data, and obtain the first data.

[0065] The following will further explain and illustrate steps S510 and S520. Specifically, intermediate data is categorized using Flink's key-based differentiation operator, i.e., keyBy. This categorization can be performed by classifying the data based on the server cluster to which the process belongs, resulting in categorized data, which still contains intermediate data. For each categorized data entry, its key can be obtained. Based on the reporting timestamp within that key, it's determined whether the data is expired. If expired data is identified, since the intermediate data stores the first target data, this past first target data can be directly deleted from the target data, yielding the first data. Expired data is defined as data whose reporting timestamp of the current process is earlier than or equal to the reporting timestamp of a previous process.

[0066] For example, if a process reports data at 12:00:00, during the next data parsing, it is found that the process reported data with a timestamp no greater than 12:00:00, which is expired data.

[0067] In step S120, the first data is parsed to obtain a set of virtual objects included in the first data, and the virtual objects in the first data are aggregated according to the set of virtual objects to obtain a target set of virtual objects.

[0068] After deleting expired data from the target data to obtain the first data, this first data can be sent to Flink's StatisticParser operator. The StatisticParser operator performs the final business transformation, mapping the computed data to a "directly writable storage" model, making the data compatible with downstream storage (such as ClickHouse). That is, the original first data is converted into a flat structure compatible with ClickHouse's wide tables.

[0069] In a game, various types of virtual objects may exist, and different types of virtual objects may also exist in each region. When the process reports data for each region, different types of virtual objects are stored in different collections for reporting. Therefore, after obtaining the first data, to simplify the logic and code of Flink data processing, the nested first data can be parsed to obtain the set of virtual objects included in each piece of first data, and then the virtual object sets can be aggregated to obtain the target virtual object set. When aggregating the virtual object set, virtual objects from the same piece of first data can be aggregated into one virtual object set, or virtual objects from all pieces of first data can be aggregated into one virtual object set. In this example embodiment, no specific limitation is made.

[0070] In one exemplary embodiment, reference is made to Figure 6 As shown, the first data is parsed to obtain a set of virtual objects included in the first data. Based on this set of virtual objects, the virtual objects in the first data are aggregated to obtain a target set of virtual objects, including:

[0071] Step S610. For each piece of data in the first data, obtain the server cluster information in each piece of data, traverse the server cluster information, and obtain the process information in the server cluster information;

[0072] Step S620. Obtain the reporting data timestamp from the process information, filter the process information based on the reporting data timestamp, and obtain the target process information;

[0073] Step S630. Obtain the region information and the set of virtual objects included in the target process information, and aggregate the virtual objects in the set of virtual objects to obtain the target virtual object set.

[0074] The following will further explain and illustrate steps S610-S630. Specifically, each piece of data in the first data is parsed into a target object, which can be a Java object. After obtaining the target object, it is parsed to obtain the server cluster information within it. This server cluster information is then traversed to obtain the server cluster's key and value. The key is the server cluster number, and the value is the process information included in the server cluster information. After obtaining the process information included in the server cluster information, this process information is traversed to obtain the process's key and value. The process key is the process number, and the value is the process set, which contains region information. From the process set, the process's reported data timestamp, the server cluster to which the process belongs, the process number, the process information, and the batch of data reported by the process are obtained. This data is then passed to the SnapshotPreTimeFilter object, which filters the process information to obtain the target process information. After obtaining the target process information, it is parsed. This parsing can be performed using the `flatMap` method (automatically invoked by the Flink structure) or the `parseGameInfo` method; the specific parsing method is not limited in this example embodiment. The logic for parsing the target process information is as follows: expand the process information to obtain the number of regions, process load, and region information. Iterate through the region information to obtain the set of virtual objects included in that region. This region information can include multiple sets of virtual objects, each corresponding to a different virtual object type. After obtaining the set of virtual objects, the virtual objects included in multiple sets can be aggregated into the target virtual object set.

[0075] In one exemplary embodiment, the step of aggregating the virtual objects in the virtual object set to obtain the target virtual object set includes:

[0076] The key values ​​of the virtual objects included in the virtual object set are modified to obtain the target key value. Based on the target key value, the virtual objects included in the virtual object set are aggregated into the target virtual object set.

[0077] Specifically, since the region includes multiple types of virtual objects, the region information can include multiple virtual object collections, each corresponding to one type of virtual object. Virtual objects from different collections can be aggregated into a target virtual object collection. When aggregating different types of virtual objects, a dynamic field mapping technique is used; that is, the key value of the virtual object is modified to obtain the target key value, which can be the name of the virtual object. When modifying the name of a virtual object, a prefix can be added, which can be the type of the virtual object. After obtaining the target key value, different types of virtual objects are aggregated into the target virtual object collection based on the target key value. By aggregating different types of virtual objects into the target virtual object collection, the logic and code for Flink's data processing are greatly simplified, and system complexity is controlled.

[0078] For example, the area information may contain two sets: battleEntityMap and zoneEntityMap. These two sets contain two different types of virtual objects. The virtual objects in the variable-length battleEntityMap and zoneEntityMap are converted into a single virtual object set by adding a prefix to their names.

[0079] Furthermore, when traversing the region information, the number of virtual objects of different types can also be obtained. After aggregating virtual objects into a target virtual object set, it can be determined whether the virtual object set includes all virtual objects based on the number of virtual objects of different types.

[0080] In step S130, the data chain node associated with the virtual object in the virtual object set and the second data corresponding to the data chain node are extracted from the first data, and the data chain node and the second data are stored.

[0081] After aggregating the virtual objects to obtain the target virtual object set, for each virtual object in the target virtual object set, the data chain node associated with the virtual object can be extracted from the first data. The data chain node is the region where the virtual object is located, the process carrying the region, and the server cluster where the process is located. After obtaining the data chain node, the second data corresponding to the data chain node can be obtained from the first data, and the data chain node and the second data corresponding to the data chain node can be stored.

[0082] In one exemplary embodiment, reference is made to Figure 7As shown, the step of extracting the data chain node associated with the virtual object in the virtual object set and the second data corresponding to the data chain node from the first data, and storing the data chain node and the second data, includes:

[0083] Step S710. Obtain from the first data the data chain node corresponding to the virtual object, and the second server cluster information, second process information, and second region information corresponding to the data chain node, wherein the data chain node includes the second server cluster, second process, and second region corresponding to the virtual object;

[0084] Step S720. The second server cluster information, the second process information, the second region information, and the virtual object are determined as the second data, and the data chain node and the second data are stored.

[0085] The following will further explain and illustrate steps S710 and S720. Specifically, the data link node corresponding to the virtual object in the first data is obtained. This data link node can be the second region where the virtual object is located, the second process carrying the second region, and the second server cluster to which the second process belongs. After obtaining the data link node, the second data corresponding to the data link node can be obtained from the first data, and the data link node and the second data corresponding to it are stored. The second data corresponding to the second region is second region information, the second data corresponding to the second process is second process information, and the second data corresponding to the second server cluster is second server cluster information. The second region information may include the second region number, second region load, second region boundary, and the number and type of virtual objects included in the second region. The second process information may include the second process identifier, the number of regions carried by the second process, the server cluster to which the second process belongs, the second reported data timestamp, the IP address of the machine where the second process is located, and the second process load. The second server cluster information may include the second server cluster identifier and the number of processes included in the second server cluster.

[0086] By extracting the data chain nodes corresponding to the virtual objects in the target virtual object set and storing the data chain nodes and the second data corresponding to the data chain nodes, the relationship chain in the target data is preserved. When there is a new data display request, there is no need to modify the Flink code. Instead, only the query SQL for the new data needs to be modified to output the data corresponding to the new data display request.

[0087] For example, in related technologies, when it is necessary to add a new query dimension zoneIndex+machineIp, the Flink code must be modified and the aggregation table must be rebuilt because the relationship between the two fields is not preserved when aggregating the data. However, in this disclosure, since the complete virtual object relationship chain is preserved, only the corresponding SQL query statement needs to be modified, which simplifies the logic and code of Flink data processing and simplifies the system complexity.

[0088] Because the complete relationship chain of the original data (server cluster - process - region - virtual object) is preserved, when a fault occurs, the source can be traced based on this complete entity relationship chain, greatly shortening the fault diagnosis time. For example, when a user experiences lag during gameplay, the specific process can be located through latency logs, and the region can be quickly located by using the server cluster to which the process belongs and the logical server IDs included in the cluster. The cause of the player's lag can be determined by checking the number, type, and load of virtual objects within that region. When a server malfunctions, the faulty machine can be identified by process load; a process load increase greater than 80% indicates a machine malfunction. This optimizes maintenance manpower and reduces the frequency of maintenance intervention.

[0089] In one exemplary embodiment, after the second data is stored, the stored second data can be iterated to obtain null value fields included in the second data, and these null value fields can be deleted. Specifically, storage space can be optimized by using `entity.entrySet().removeIf(entry->entry.getValue()==0)` and removing zones without `zoneLoad` and `zoneInfo`.

[0090] In one exemplary embodiment, LZ4 connection compression can be enabled to write the stored second data to ClickHouse. During the write process, a preset number of records can be accumulated before writing, which can be 5000 records; however, this example embodiment does not specify a particular number. If a network error occurs during the write process, the system automatically retryes three times to ensure a high success rate for batch writes.

[0091] In one exemplary embodiment, the method further includes:

[0092] Obtain the timestamp of the reported data of the process corresponding to the second data, and determine whether there is duplicate data in the second data based on the timestamp of the reported data;

[0093] If duplicate data exists, delete the duplicate data.

[0094] Specifically, after storing the second data, when Flink encounters an error or crashes, it can save the second data to disk for rereading the data before the crash after recovery. When saving the second data to disk, a Long parameter can be used to record the timestamp of the process that reported the second data for each piece of data. The timestamp of the process's reported data is used to determine whether there is duplicate data in the recovered second data. If duplicate data is found, it is deleted.

[0095] Furthermore, the data expiration policy can be 24 hours. That is, if the interval between the current time and the timestamp of the data reported by the process corresponding to the virtual object data exceeds 24 hours, the virtual object data will be automatically deleted to prevent the virtual object data from growing indefinitely. This expiration policy reduces memory usage and also shortens job recovery time.

[0096] For example, after data recovery, if a process reports data at 12:00:00, and then during the next data parsing, it is found that the process reported data with a timestamp no greater than 12:00:00, this data is judged as duplicate data.

[0097] The data processing method provided by the exemplary embodiments of this disclosure has at least the following advantages: On the one hand, after obtaining the target data, the target data is converted into intermediate data, and the expired data in the target data is determined using the intermediate data and deleted, thereby reducing memory usage; on the other hand, after deleting the expired data, the first data is parsed to obtain a set of virtual objects included in the first data, and the virtual objects are aggregated according to the set of virtual objects, except that different types of virtual objects are aggregated into a table. For each virtual object in the target set of virtual objects, the data chain node corresponding to each virtual object is determined in the first data, and the data chain node is associated with the corresponding data chain node. The second data is extracted and stored. Specifically, data chain nodes are extracted and their corresponding second data are stored. The target data corresponding to the virtual object is not aggregated, preserving the data chain associated with the virtual object within the target data. This avoids destroying the multidimensional association information in the target data, resolving the issue of premature data aggregation in related technologies that leads to the destruction of multidimensional association information in the original target data. This simplifies Flink's data processing logic and code, reducing system complexity. Furthermore, when new data display requests arise, no modification to the Flink code is required; only the SQL statement executed by the query backend needs to be modified, improving data processing efficiency.

[0098] Exemplary embodiments of this disclosure also provide a data processing apparatus, with reference to Figure 8 As shown, the data processing device includes the following:

[0099] The target data acquisition module 810 is used to acquire target data based on a data acquisition request, convert the target data into intermediate data, and filter the target data based on the intermediate data to obtain first data;

[0100] The virtual object aggregation module 820 is used to parse the first data to obtain a set of virtual objects included in the first data, and to aggregate the virtual objects in the first data according to the set of virtual objects to obtain a target set of virtual objects;

[0101] The data storage module 830 is used to extract data chain nodes associated with virtual objects in the virtual object set and second data corresponding to the data chain nodes from the first data, and to store the data chain nodes and the second data.

[0102] In one exemplary embodiment, the target data acquisition module includes:

[0103] The request sending module is used to send a data retrieval request to the microservice cluster, and to retrieve and save the first page of data and the total number of pages sent by the microservice cluster.

[0104] The remaining data request module is used to send remaining data retrieval requests to the microservice cluster in parallel and to receive remaining page data sent by the microservice cluster.

[0105] The data splicing module is used to splice the first page data with the remaining page data to obtain spliced ​​data, and to decompress the spliced ​​data to obtain the target data.

[0106] In one exemplary embodiment, the target data includes at least: server cluster information, process information included in the server cluster information, region information included in the process information, and a set of virtual objects included in the region information.

[0107] In one exemplary embodiment, the target data acquisition module includes:

[0108] The process information acquisition module is used to acquire server cluster information in the first target data of the target data, and to traverse the server cluster information to obtain the process information included in the server cluster information.

[0109] The intermediate state data acquisition module is used to acquire the first attribute data in the process information, use the first attribute data as a key and the first target data as a value, and store the key-value pairs to obtain the intermediate state data.

[0110] In one exemplary embodiment, the first attribute data includes at least: process identifier, server cluster to which the process belongs, and timestamp of the data reported by the process.

[0111] In one exemplary embodiment, the target data acquisition module includes:

[0112] The data classification module is used to classify the intermediate data based on the process identifier in the first attribute data and the server cluster to which the process belongs, using a key-based differentiation operator, to obtain classified data.

[0113] The expired data deletion module is used to determine the expired data included in the classified data based on the timestamp of the data reported by the process, delete the expired data in the target data, and obtain the first data.

[0114] In one exemplary embodiment, the virtual object aggregation module includes:

[0115] The process information acquisition module is used to acquire server cluster information in each piece of data in the first data, traverse the server cluster information, and obtain process information in the server cluster information.

[0116] The process filtering module is used to obtain the reporting data timestamp in the process information, filter the process information based on the reporting data timestamp, and obtain the target process information.

[0117] The aggregation module uses the region information and the set of virtual objects included in the target process information to aggregate the virtual objects in the set of virtual objects to obtain the target set of virtual objects.

[0118] In one exemplary embodiment, the aggregation module includes:

[0119] The key-value modification module is used to modify the key values ​​of the virtual objects included in the virtual object set to obtain the target key value, and to aggregate the virtual objects included in the virtual object set into the target virtual object set based on the target key value.

[0120] In one exemplary embodiment, the data storage module includes:

[0121] The data chain node extraction module is used to obtain the data chain node corresponding to the virtual object from the first data, as well as the second server cluster information, second process information, and second region information corresponding to the data chain node, wherein the data chain node includes the second server cluster, second process, and second region corresponding to the virtual object;

[0122] The second data storage module is used to determine the second server cluster information, the second process information, the second region information, and the virtual object as the second data, and to store the data chain node and the second data.

[0123] In one exemplary embodiment, the data storage module includes:

[0124] The duplicate data determination module is used to obtain the reporting data timestamp of the process corresponding to the second data, and determine whether there is duplicate data in the second data based on the reporting data timestamp;

[0125] The deduplication module is used to delete the duplicate data when duplicate data exists.

[0126] The specific details of each part of the above-mentioned device have been described in detail in the method section of the implementation plan. For any undisclosed details, please refer to the implementation plan of the method section, and therefore will not be repeated here.

[0127] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to exemplary embodiments of this disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0128] Furthermore, although the steps of the method in this invention are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in that specific order, or that all the steps shown must be performed to achieve the desired result. Additional or alternative steps may be omitted, multiple steps may be combined into one step, and / or a step may be broken down into multiple steps.

[0129] In an exemplary embodiment of the present invention, an electronic device capable of implementing the above-described method is also provided.

[0130] Those skilled in the art will understand that various aspects of the present invention can be implemented as systems, methods, or program products. Therefore, various aspects of the present invention can be specifically implemented in the following forms: entirely hardware implementations, entirely software implementations (including firmware, microcode, etc.), or implementations combining hardware and software aspects, collectively referred to herein as “circuits,” “modules,” or “systems.”

[0131] The following reference Figure 9 To describe an electronic device 900 according to this embodiment of the present invention. Figure 9The electronic device 900 shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.

[0132] like Figure 9 As shown, the electronic device 900 is manifested in the form of a general-purpose computing device. The components of the electronic device 900 may include, but are not limited to: at least one processing unit 910, at least one storage unit 920, a bus 930 connecting different system components (including storage unit 920 and processing unit 910), and a display unit 940.

[0133] The storage unit stores program code that can be executed by the processing unit 910, causing the processing unit 910 to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of the present invention. For example, the processing unit 910 can perform actions such as... Figure 1 Step S110: Based on the data acquisition request, acquire target data, convert the target data into intermediate data, filter the target data based on the intermediate data to obtain first data; Step S120: Parse the first data to obtain a set of virtual objects included in the first data, aggregate the virtual objects in the first data according to the set of virtual objects to obtain a target set of virtual objects; Step S130: Extract the data chain node associated with the virtual objects in the set of virtual objects and the second data corresponding to the data chain node from the first data, and store the data chain node and the second data.

[0134] Storage unit 920 may include readable media in the form of volatile storage units, such as random access memory (RAM) 9201 and / or cache memory 9202, and may further include read-only memory (ROM) 9203.

[0135] Storage unit 920 may also include a program / utility 9204 having a set (at least one) program module 9205, such program module 9205 including but not limited to: operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0136] Bus 930 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the various bus structures.

[0137] Electronic device 900 can also communicate with one or more external devices 1000 (e.g., keyboard, pointing device, Bluetooth device, etc.), and with one or more devices that enable a user to interact with electronic device 900, and / or with any device that enables electronic device 900 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 950. Furthermore, electronic device 900 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 960. As shown, network adapter 960 communicates with other modules of electronic device 900 via bus 930. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 900, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID (Redundant Arrays of Independent Disks) systems, tape drives, and data backup storage systems.

[0138] Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of the present invention can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, portable hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, terminal device, or network device, etc.) to execute the method according to the embodiments of the present invention.

[0139] In exemplary embodiments of the present invention, a computer-readable storage medium is also provided, on which a program product capable of implementing the methods described above is stored. In some possible embodiments, various aspects of the present invention may also be implemented as a program product comprising program code that, when the program product is run on a terminal device, causes the terminal device to perform the steps of the various exemplary embodiments of the present invention described in the "Exemplary Methods" section above.

[0140] According to embodiments of the present invention, a program product for implementing the above-described method may employ a portable compact disc read-only memory (CD-ROM) and include program code, and may run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto. In this document, a readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0141] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0142] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting programs for use by or in conjunction with an instruction execution system, apparatus, or device.

[0143] The program code contained on the readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF (Radio Frequency), etc., or any suitable combination thereof.

[0144] Program code for performing the operations of this invention can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0145] Furthermore, the above figures are merely illustrative of the processes included in the method according to exemplary embodiments of the present invention, and are not intended to be limiting. It is readily understood that the processes shown in the above figures do not indicate or limit the temporal order of these processes. Additionally, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0146] Other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention described herein. This application is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not invented herein. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of the invention are indicated by the claims.

Claims

1. A data processing method, characterized in that, include: Based on the data acquisition request, target data is acquired, the target data is converted into intermediate data, and the target data is filtered based on the intermediate data to obtain first data; The first data is parsed to obtain a set of virtual objects included in the first data. The virtual objects in the first data are aggregated according to the set of virtual objects to obtain a target set of virtual objects. Extract the data chain node associated with the virtual object in the virtual object set and the second data corresponding to the data chain node from the first data, and store the data chain node and the second data.

2. The method according to claim 1, characterized in that, The process of obtaining target data based on the data acquisition request includes: Send a data retrieval request to the microservice cluster, retrieve and save the first page of data sent by the microservice cluster and the total number of pages; Send remaining data retrieval requests to the microservice cluster in parallel, and receive remaining page data sent by the microservice cluster; The data from the first page is concatenated with the data from the remaining pages to obtain concatenated data. The concatenated data is then decompressed to obtain the target data.

3. The method according to claim 1, characterized in that, The target data includes at least: server cluster information, process information included in the server cluster information, region information included in the process information, and a set of virtual objects included in the region information.

4. The method according to claim 3, characterized in that, The process of converting the target data into intermediate data includes: For the first target data in the target data, obtain the server cluster information in the first target data, and traverse the server cluster information to obtain the process information included in the server cluster information; Obtain the first attribute data from the process information, use the first attribute data as the key and the first target data as the value, store the key-value pair to obtain the intermediate state data.

5. The method according to claim 4, characterized in that, The first attribute data includes at least: process identifier, server cluster to which the process belongs, and timestamp of the data reported by the process.

6. The method according to claim 5, characterized in that, The step of filtering the target data based on the intermediate state data to obtain the first data includes: Using a key-based differentiation operator, the intermediate data is classified according to the process identifier in the first attribute data and the server cluster to which the process belongs, to obtain classified data; Based on the timestamp of the data reported by the process, the expired data included in the classified data is determined, and the expired data is deleted from the target data to obtain the first data.

7. The method according to claim 3, characterized in that, The first data is parsed to obtain a set of virtual objects included in the first data. The virtual objects in the first data are then aggregated based on the set of virtual objects to obtain a target set of virtual objects, including: For each piece of data in the first data, obtain the server cluster information in each piece of data, traverse the server cluster information, and obtain the process information in the server cluster information; Obtain the reporting data timestamp from the process information, filter the process information based on the reporting data timestamp, and obtain the target process information; Obtain the region information and the set of virtual objects included in the target process information, and aggregate the virtual objects in the set of virtual objects to obtain the target virtual object set.

8. The method according to claim 7, characterized in that, The aggregation of virtual objects in the virtual object set to obtain the target virtual object set includes: The key values ​​of the virtual objects included in the virtual object set are modified to obtain the target key value. Based on the target key value, the virtual objects included in the virtual object set are aggregated into the target virtual object set.

9. The method according to claim 1, characterized in that, The step of extracting the data chain node associated with the virtual object in the virtual object set and the second data corresponding to the data chain node from the first data, and storing the data chain node and the second data, includes: Obtain from the first data the data chain node corresponding to the virtual object, and the second server cluster information, second process information, and second region information corresponding to the data chain node, wherein the data chain node includes the second server cluster, second process, and second region corresponding to the virtual object; The second server cluster information, the second process information, the second region information, and the virtual object are identified as the second data, and the data chain node and the second data are stored.

10. The method according to claim 1, characterized in that, The method further includes: Obtain the timestamp of the reported data of the process corresponding to the second data, and determine whether there is duplicate data in the second data based on the timestamp of the reported data; If duplicate data exists, delete the duplicate data.

11. A data processing apparatus, characterized in that, include: The target data acquisition module is used to acquire target data based on a data acquisition request, convert the target data into intermediate data, and filter the target data based on the intermediate data to obtain first data; The virtual object aggregation module is used to parse the first data to obtain a set of virtual objects included in the first data, and to aggregate the virtual objects in the first data according to the set of virtual objects to obtain a target set of virtual objects; The data storage module is used to extract data chain nodes associated with virtual objects in the virtual object set and second data corresponding to the data chain nodes from the first data, and to store the data chain nodes and the second data.

12. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method according to any one of claims 1 to 10.

13. An electronic device, characterized in that, include: processor; Memory for storing the executable instructions of the processor; The processor is configured to execute the method of any one of claims 1 to 10 by executing the executable instructions.

Images