Data processing method and device, computer equipment, storage medium and program product
By detecting the primary key of the data and aggregating or storing it independently, the storage bottleneck problem of the Prometheus monitoring system was solved, enabling efficient remote data storage and querying, and simplifying the data processing flow.
Patent Information
- Application Number
- CN202511776013.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-28
- Publication Date
- 2026-03-06
AI Technical Summary
The Prometheus monitoring system suffers from storage space and performance bottlenecks when handling large-scale, long-term data storage, and the process of remote data storage is complex.
By receiving data storage requests, the system detects whether there are identical primary keys in the aggregated data table, and stores the data according to the detection results, including aggregate processing or independent storage. It also uses a reverse index table for data tag storage and querying, simplifying the remote storage process.
It reduces redundant data storage, improves storage efficiency and data consistency, simplifies remote storage operations, and reduces maintenance costs and query complexity.
Smart Images
Figure CN121614342A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of data storage technology, and in particular to a data processing method, apparatus, computer equipment, storage medium, and program product. Background Technology
[0002] Prometheus, an open-source time-series database monitoring system, has been widely used in scenarios such as cloud monitoring and system performance monitoring. Its local storage is suitable for short-term data storage and real-time monitoring, but when dealing with large-scale, long-term data storage, it often encounters storage space and performance bottlenecks. Therefore, remote storage technology is used to transfer the data monitored by Prometheus to an external storage system to alleviate the pressure on local storage.
[0003] However, the above methods have the problem of complex remote data storage processes. Summary of the Invention
[0004] Therefore, it is necessary to provide a data processing method, apparatus, computer equipment, storage medium, and program product that can simplify the process of remote data storage in response to the above-mentioned technical problems.
[0005] Firstly, this application provides a data processing method for a first server. The method includes:
[0006] Receive a data storage request sent by a second server, wherein the data storage request includes data to be stored and a first data primary key of the data to be stored;
[0007] Based on the first data primary key, detect whether there is a target data primary key that is the same as the first data primary key among the multiple second data primary keys included in the aggregated data table, and obtain the detection result;
[0008] Based on the detection results, the data to be stored is stored in the aggregated data table, and a response message is sent to the second server, the response message indicating that the first server has stored the data to be stored.
[0009] In one embodiment, the step of storing the data to be stored in the aggregated data table based on the detection result and sending a response message to the second server, the response message indicating that the first server has stored the data to be stored, includes:
[0010] If the detection result indicates that the target data primary key exists among the plurality of second data primary keys, then the data to be stored is aggregated to obtain aggregated data, and the aggregated data is stored in the aggregated data table;
[0011] If the detection result indicates that the target data primary key does not exist among the plurality of second data primary keys, then the data to be stored is independently stored in the aggregated data table.
[0012] In one embodiment, the step of aggregating the data to be stored to obtain aggregated data and storing the aggregated data in the aggregated data table includes:
[0013] Obtain the aggregation type corresponding to the primary key of the target data;
[0014] According to the aggregation rules corresponding to the aggregation type, the data to be stored and the historical data corresponding to the primary key of the target data are aggregated to obtain the aggregated data.
[0015] The historical data stored in the aggregated data table is replaced with the aggregated data.
[0016] In one embodiment, the data storage request further includes at least one first data tag, and the method further includes:
[0017] The first data primary key and the at least one first data tag are stored in a tag index table using a reverse index method. The tag index table includes multiple sets of correspondences, including the correspondence between the data primary key and at least one data tag.
[0018] In one embodiment, the method further includes:
[0019] Receive a data query request sent by the second server, wherein the data query request includes a second data tag;
[0020] Based on the second data tag and the tag index table, obtain the data to be queried corresponding to the data query request.
[0021] In one embodiment, the number of second data tags is multiple, and the step of obtaining the query data corresponding to the data query request based on the second data tags and the tag index table includes:
[0022] Multiple target data tags are determined from the tag index table based on multiple second data tags, wherein the multiple target data tags are at least partially identical to the multiple second data tags;
[0023] The data primary key corresponding to each of the target data tags is determined as a set of data primary keys;
[0024] The data to be queried is obtained from the aggregated data table based on the data primary key set, and the data to be queried is sent to the second server.
[0025] Secondly, this application also provides a data processing apparatus for a first server. The apparatus includes:
[0026] The first receiving module is used to receive a data storage request sent by the second server, wherein the data storage request includes data to be stored and a first data primary key of the data to be stored;
[0027] The detection module is used to detect, based on the first data primary key, whether there is a target data primary key that is the same as the first data primary key among the multiple second data primary keys included in the aggregated data table, and to obtain the detection result;
[0028] The first storage module is used to store the data to be stored in the aggregated data table according to the detection result, and send a response message to the second server, the response message being used to indicate that the first server has stored the data to be stored.
[0029] Thirdly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps of the method described in the first aspect.
[0030] Fourthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, implements the steps of the method described in the first aspect.
[0031] Fifthly, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, implements the steps of the method described in the first aspect.
[0032] The aforementioned data processing method, apparatus, computer equipment, storage medium, and program product involve a first server receiving a data storage request from a second server. The data storage request includes the data to be stored and a first primary key for the data to be stored. Then, based on the first primary key, the system checks whether a target primary key identical to the first primary key exists among the multiple second primary keys included in the aggregated data table, obtaining a detection result. Based on the detection result, the data to be stored is stored in the aggregated data table, and a response message is sent to the second server. The response message indicates that the first server has stored the data to be stored. In this way, by detecting the existence of a target primary key, no additional redundant verification process is needed, preventing multiple data with the same primary key from existing in the aggregated data table, thus avoiding data redundancy and wasting storage space. Data storage based on the detection result reduces invalid data storage operations without complex configuration or collaborative processing, ensuring the consistency and accuracy of data in the aggregated data table while simplifying the remote, off-site data storage process. Attached Figure Description
[0033] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.
[0034] Figure 1 This is a diagram illustrating the application environment of a data processing method in one embodiment.
[0035] Figure 2 This is a flowchart illustrating a data processing method in one embodiment;
[0036] Figure 3 This is a flowchart illustrating step 203 in one embodiment;
[0037] Figure 4 This is a flowchart illustrating step 301 in one embodiment;
[0038] Figure 5 This is a flowchart illustrating the data processing method in another embodiment;
[0039] Figure 6 This is a flowchart illustrating step 502 in one embodiment;
[0040] Figure 7 This is a structural block diagram of a data processing device in one embodiment;
[0041] Figure 8 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation
[0042] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.
[0043] The data processing method provided in this application embodiment can be applied to, for example... Figure 1 In the application environment shown, the first server 102 communicates with the second server 104 via a network. The first data storage system can store the data that the first server 102 needs to process. The first data storage system can be integrated onto the first server 102 or placed on a cloud or other network server. The second data storage system can store the data that the second server 104 needs to process. The second data storage system can be integrated onto the second server 104 or placed on a cloud or other network server. The first server 102 first receives a data storage request sent by the second server 104. The data storage request includes the data to be stored and a first primary key of the data to be stored. Then, based on the first primary key, it checks whether there is a target primary key with the same name as the first primary key among the multiple second primary keys included in the aggregated data table, obtaining the detection result. Then, based on the detection result, it stores the data to be stored in the aggregated data table and sends a response message to the second server, indicating that the first server has stored the data to be stored. The first server 102 and the second server 104 can be implemented using independent servers or a server cluster composed of multiple servers.
[0044] In one exemplary embodiment, such as Figure 2 As shown, a data processing method is provided, which can be applied to... Figure 1 Taking the first server 102 as an example, the explanation includes the following steps:
[0045] Step 201: Receive a data storage request sent by the second server.
[0046] The data storage request includes the data to be stored and the primary key of the data to be stored.
[0047] The primary key is an identifier used to uniquely identify the data to be stored, ensuring the uniqueness and integrity of the data.
[0048] It should be noted that time-series database monitoring systems generate a large amount of data. Storing all this data on the local server where the monitoring system is deployed would result in storage space and performance bottlenecks. Therefore, this type of data can be stored in other remote storage. In this embodiment, the local server can be used as the second server, and the remote server as the first server. Taking the first server as an example, the process of remote data storage will be described exemplarily.
[0049] It is understandable that a data storage request can be a request that generates new data as the data to be stored, in the case of new data being generated on a second server, and that determines the first primary key of the data to be stored.
[0050] In this embodiment, the first server can receive data storage requests sent by the second server via the network.
[0051] In one possible implementation, the first server can parse the data storage request to obtain the data to be stored and the primary key of the data to be stored.
[0052] Step 202: Based on the first data primary key, detect whether there is a target data primary key that is the same as the first data primary key among the multiple second data primary keys included in the aggregated data table, and obtain the detection result.
[0053] The aggregation data table is a data table on the first server used to store aggregated data, which is the data obtained after aggregating the original data. Optionally, the aggregation data table can be a data table in a database, for example, the database can be a Doris analytical database.
[0054] The second primary key is a unique identifier for each piece of data stored in the aggregated data table.
[0055] Understandably, before storing the data to be stored on the first server, it's possible to check if historical data exists in the aggregated data table that can be aggregated with the data to be stored. To ensure the accuracy of the detection results, it can be done based on the primary key. In this embodiment, the primary key in the aggregated data table that is the same as the first primary key is determined as the target primary key.
[0056] Optionally, the detection result may include the existence of a target data primary key among the multiple second data primary keys included in the aggregated data table; or, the absence of a target data primary key among the multiple second data primary keys included in the aggregated data table.
[0057] In this embodiment, the first server can compare the first data primary key with each of the second data primary keys. If the comparison result indicates that there is a data primary key that is the same as the first data primary key, then the data primary key of the data primary key is determined as the target primary key, and a detection result is obtained that there is a target data primary key among the multiple second data primary keys included in the aggregated data table; if the comparison result indicates that there is no data primary key that is the same as the first data primary key, then a detection result is obtained that there is no target data primary key among the multiple second data primary keys included in the aggregated data table.
[0058] Step 203: Based on the detection results, store the data to be stored in the aggregated data table and send a response message to the second server.
[0059] The response message is used to indicate that the first server has stored the data to be stored.
[0060] It should be noted that, in order to ensure data synchronization, after the first server successfully stores the data to be stored, it can send a notification message to the second server. In this way, the second server can confirm that the data to be stored has been successfully stored in the first server, and thus can delete the stored original data.
[0061] Understandably, different detection results may necessitate different storage methods for the data to be stored. For example, if a target primary key exists, the data can be stored using the first storage method, updating the data as needed. If no target primary key exists, the data can be stored using the second storage method, inserting the data as needed.
[0062] In this embodiment, the first server can determine the storage method based on the detection results, store the data to be stored in the aggregation table according to the determined storage method, generate a corresponding response message, and then send the response message to the first server through the network.
[0063] In the above data processing method, the first server first receives a data storage request sent by the second server. The data storage request includes the data to be stored and a first primary key of the data to be stored. Then, based on the first primary key, it checks whether there is a target primary key with the same first primary key among the multiple second primary keys included in the aggregated data table, and obtains the detection result. Then, based on the detection result, the data to be stored is stored into the aggregated data table, and a response message is sent to the second server. The response message is used to indicate that the first server has stored the data to be stored. In this way, by checking whether there is a target primary key, no additional redundant verification process is needed, which can prevent multiple data with the same primary key from existing in the aggregated data table, thus preventing data redundancy and wasting extra storage space. Therefore, data storage is performed based on the detection result, without complex configuration or collaborative processing, which can reduce invalid data storage operations. While ensuring the consistency and accuracy of the data in the aggregated data table, it simplifies the storage process of remote off-site data storage.
[0064] In one exemplary embodiment, such as Figure 3 As shown, this embodiment relates to the process by which the first server stores the data to be stored in the aggregated data table based on the detection results and sends a response message to the second server. Step 203 includes:
[0065] Step 301: If the detection result shows that the target data primary key exists among multiple second data primary keys, then the data to be stored is aggregated to obtain aggregated data, and the aggregated data is stored in the aggregated data table.
[0066] Aggregation processing refers to the process of merging and calculating multiple data entries to integrate them into a single data entry. Aggregated data refers to the data obtained after aggregating at least two data entries. For example, if the data to be stored is CPU utilization monitoring data, and the target data primary key exists in the aggregated data table, multiple CPU utilization monitoring data entries within one hour can be merged into a single aggregated data entry.
[0067] In this embodiment, if the detection result shows that the target data primary key exists among multiple second data primary keys, the first server can aggregate the data to be stored with the data corresponding to the target data primary key, determine the aggregated result as aggregated data, and then store the aggregated data in the aggregated data table.
[0068] Step 302: If the detection result shows that the target data primary key does not exist among the multiple second data primary keys, then the data to be stored is stored independently in the aggregated data table.
[0069] Understandably, if the target data primary key does not exist, it means that there is no need to aggregate the data to be stored with the existing historical data with the same primary key in the aggregate data table. In this case, the data to be stored can be inserted into the aggregate data table as an independent, new record.
[0070] In this embodiment, if the first server detects that the target data primary key does not exist among multiple second data primary keys, it can independently store the data to be stored in the aggregated data table by inserting data.
[0071] In this embodiment, the first server aggregates the data to be stored to obtain aggregated data when the detection result shows that the target data primary key exists among multiple second data primary keys, and stores the aggregated data in the aggregated data table. Conversely, when the detection result shows that the target data primary key does not exist among multiple second data primary keys, the data to be stored is stored independently in the aggregated data table. In this way, data redundancy can be avoided by performing aggregation processing before storage, and new data can be stored quickly and completely by independent storage. This achieves the dual task of deduplication aggregation and compatibility with new data addition. Thus, the maintenance cost can be reduced and the structure and availability of data in the aggregated data table can be guaranteed through the logic of automatic aggregation or storage.
[0072] In one exemplary embodiment, such as Figure 4 As shown, this embodiment relates to the process by which the first server aggregates the data to be stored to obtain aggregated data and stores the aggregated data in the aggregated data table. Step 301 above includes:
[0073] Step 401: Obtain the aggregation type corresponding to the primary key of the target data.
[0074] The aggregation type refers to the type of data aggregation processing performed. Optionally, the aggregation type can be maximum value aggregation, minimum value aggregation, sum aggregation, mean aggregation, periodic aggregation, etc.
[0075] In this embodiment, the first server can obtain the aggregation type corresponding to the primary key of the target data from the records of the metadata of the aggregation data table.
[0076] Step 402: According to the aggregation rules corresponding to the aggregation type, aggregate the historical data corresponding to the primary key of the data to be stored and the target data to obtain aggregated data.
[0077] The aggregation rule is the specific logic for merging new data and historical data based on the aggregation type. Historical data refers to the data corresponding to the primary key of the target data stored in the aggregation table.
[0078] Understandably, different aggregation types correspond to different aggregation rules. For example, the aggregation rule for the maximum value aggregation type is to use the larger value between the new data and the historical data as the aggregated data; the aggregation rule for the sum aggregation type is to use the sum of the new data and the historical data as the aggregated data.
[0079] In this embodiment, the first server can merge the data to be stored and the historical data according to the aggregation rules of the aggregation type corresponding to the primary key of the target data to obtain aggregated data.
[0080] Step 403: Replace the historical data stored in the aggregated data table with the aggregated data.
[0081] Understandably, when new aggregated data is generated, in order to avoid storing redundant data in the aggregated data table, the aggregated data can be stored using an update storage method.
[0082] In this embodiment, the first server can delete the historical data corresponding to the target data primary key stored in the aggregated data table, and then store the aggregated data and the target data primary key in the aggregated data table.
[0083] In this embodiment, the first server first obtains the aggregation type corresponding to the primary key of the target data, and then performs aggregation processing on the data to be stored and the historical data corresponding to the primary key of the target data according to the aggregation rules corresponding to the aggregation type to obtain aggregated data. Then, the aggregated data is used to replace the historical data stored in the aggregated data table. This ensures that the data with the same primary key in the aggregated data table is always the latest aggregation result calculated by unified rules, avoiding data redundancy. It also saves the tedious operation of fully traversing the original data through incremental aggregation mode, which greatly improves data update efficiency and storage resource utilization.
[0084] In an exemplary embodiment, the data storage request further includes at least one first data tag. This embodiment relates to the process of how a first server stores the first data tag. The above method further includes:
[0085] Using a reverse index, store the first data primary key and at least one first data tag in the tag index table.
[0086] The tag index table includes multiple sets of correspondences, each relating a primary key to at least one data tag. The tag index table is a structured table used to store tag-based indexes. It can include multiple sets of correspondences between primary keys and data tags. Optionally, each set of correspondences can include a primary key and a data tag, or a primary key and multiple data tags.
[0087] The first data tag is the data tag corresponding to the data to be stored. Optionally, the first data tag may include one or more. For example, the first data tag may include: monitoring indicators, device identifiers, time granularity, and region.
[0088] An inverted index is a reverse-mapping index storage structure used to represent the mapping relationship between data primary keys, data tags, data storage locations, or related information. It's important to note that inverted indexes can be used in multi-dimensional tag query scenarios. By storing the mapping relationship between each tag and all corresponding data locations, the query results can be obtained by calculating the intersection of multiple indexes when performing multi-condition queries, without needing to query all the original data. Therefore, inverted indexes can improve query efficiency under multiple query conditions.
[0089] For example, the data pairs stored in the tag index table can be in the form of: index key-data location, where the index key can be a first data primary key and / or at least one first data tag.
[0090] In this embodiment, the first server obtains the storage location of the data to be stored, then establishes a data pair between the first data primary key and at least one first data tag and the storage location, and then inserts the data pair into the primary key mapping column and the tag mapping column corresponding to the tag index table.
[0091] In this embodiment, the first server stores the first data primary key and at least one first data tag in the tag index table using a reverse index, thus establishing a mapping relationship between the data primary key and the data tag. In this way, during subsequent data queries, it is not necessary to scan the entire aggregated data table. The target data can be quickly located through the mapping relationship in the data index table, thereby improving data query efficiency. Moreover, since the reverse index method can support single-tag index or multi-tag index simultaneously, it can adapt to complex business query requests without the need for data filtering or splicing, reducing the complexity of the query logic.
[0092] In one exemplary embodiment, such as Figure 5 As shown, this embodiment relates to the process of how the first server responds to a data query request. The above method also includes:
[0093] Step 501: Receive a data query request sent by the second server.
[0094] The data query request includes a second data tag. The second data tag is the data tag of the data to be queried corresponding to the data query request.
[0095] Among them, the data query request is an instruction used to instruct the first server to perform a data query.
[0096] Optionally, the data query request can be sent by another device to the second server, which then forwards it to the first server; alternatively, the data query request can be generated and sent by the second server.
[0097] In this embodiment, the first server can receive a data query request sent by the second server via the network, and by parsing the data query request, the second data tag included in the data query request can be obtained.
[0098] Step 502: Obtain the data to be queried corresponding to the data query request based on the second data label and the label index table.
[0099] In this embodiment, the first server can look up the second data tag in the tag index table, then determine the storage location of the data to be queried corresponding to the data query request based on the association information corresponding to the second data tag, and then read the data to be queried from the storage location.
[0100] In this embodiment, the first server receives a data query request sent by the second server. The data query request includes a second data tag. Then, based on the second data tag and the tag index table, the server obtains the data to be queried corresponding to the data query request. The server can quickly locate the data storage location by relying on the tag index table, thereby improving the query response efficiency. Moreover, during the query process, the server can query data based on the direct association between the second data tag and the tag index table, thereby reducing the query complexity and resource consumption.
[0101] In one exemplary embodiment, the number of second data tags is multiple, such as... Figure 6 As shown, this embodiment relates to the process by which the first server obtains the data to be queried corresponding to the data query request based on the second data tag and the tag index table. Step 502 above includes:
[0102] Step 601: Determine multiple target data labels from the label index table based on multiple second data labels.
[0103] In this context, multiple target data labels are at least partially identical to multiple sub-labels. The target data label is the same as the second data label in the label index table.
[0104] It is understandable that the tag index table may contain one or more data tags that are identical to the second data tag. In this embodiment, any data tag in the tag index table that is identical to any of the second data tags is identified as the target data tag.
[0105] In this embodiment, the first server can compare the second data tag with each data tag in the tag index table, and determine the data tag that matches the comparison result as the target data tag.
[0106] Step 602: Determine the data primary key set corresponding to each target data tag.
[0107] It should be noted that each target data tag may correspond to one primary key or multiple primary keys. Therefore, data can be retrieved from the aggregated data table based on all primary keys corresponding to all target data tags. To reduce the number of data queries, all primary keys can be merged into a single set, and data can be retrieved based on this set.
[0108] Understandably, during the merging process, in order to ensure that each primary key in the final set of primary keys satisfies the conditions of all target data tags, an intersection operation can be performed on the primary keys corresponding to each target data tag.
[0109] In this embodiment, the first server can first collect all the data primary keys corresponding to each target data tag to obtain a sub-data set, then perform an intersection operation on each sub-data set, and finally determine the data primary key set by the result of the intersection operation.
[0110] Step 603: Obtain the data to be queried from the aggregated data table based on the data primary key set, and send the data to be queried to the second server.
[0111] In this embodiment, the first server can use the set of data primary keys as filtering conditions to perform multi-condition queries in the aggregated data table. Then, it can obtain the data corresponding to the data primary keys that meet the filtering conditions in the aggregated data table, use this data as the data to be queried, and then send the data to be queried to the second server through the network.
[0112] In this embodiment, the first server first determines multiple target data tags from the tag index table based on multiple second data tags. These multiple target data tags are at least partially the same as the multiple second data tags. This way, relevant data can be hit without completely identical tags, improving query performance. Then, the primary key corresponding to each target data tag is determined as a set of primary keys. The data to be queried is obtained from the aggregated data table based on the set of primary keys. This allows for accurate querying of aggregated data based on tags and primary keys, avoiding full table scans and improving query response speed. Finally, the data to be queried is sent to the second server to provide timely and reliable data support for subsequent business processing and optimize cross-server data query performance.
[0113] To facilitate understanding by those skilled in the art, the data processing method provided in this application will be described in detail below. This method may include:
[0114] S1, the second server (also known as the local server) sends a data storage request to the first server (also known as the remote server).
[0115] The data storage request includes the data to be stored and the primary key of the data to be stored.
[0116] S2, the first server receives the data storage request.
[0117] S3, the first server checks whether there is a target data primary key that is the same as the first data primary key among the multiple second data primary keys included in the aggregated data table, based on the first data primary key, and obtains the detection result.
[0118] S4. If the detection result shows that the target data primary key exists among multiple second data primary keys, then the first server obtains the aggregation type corresponding to the target data primary key.
[0119] S5, the first server performs aggregation processing on the data to be stored and the historical data corresponding to the primary key of the target data according to the aggregation rules corresponding to the aggregation type, and obtains the aggregated data.
[0120] S6, the first server uses aggregated data to replace the historical data stored in the aggregated data table.
[0121] S7. If the detection result is that the target data primary key does not exist among the multiple second data primary keys, the first server will independently store the data to be stored in the aggregate data table.
[0122] S8, the first server stores the first data primary key and at least one first data tag in the tag index table in reverse order.
[0123] The tag index table includes multiple sets of correspondences, including the correspondence between the data primary key and at least one data tag.
[0124] S9, the second server receives the data query request and sends the data query request to the first server.
[0125] The data query request includes a second data tag.
[0126] S10, the first server receives a data query request sent by the second server.
[0127] S11, the first server determines multiple target data tags from the tag index table based on multiple second data tags.
[0128] Among them, multiple target data labels are at least partially identical to multiple second data labels.
[0129] S12, the first server determines the data primary key set corresponding to each target data tag.
[0130] S13, the first server retrieves the data to be queried from the aggregated data table based on the data primary key set, and sends the data to be queried to the second server.
[0131] S14, the first server sends the data to be queried to the second server.
[0132] It should be noted that the descriptions in S1-S14 above can be found in the relevant descriptions in the above embodiments, and their effects are similar, so they will not be repeated here.
[0133] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.
[0134] Based on the same inventive concept, this application also provides a data processing apparatus for implementing the data processing method described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more data processing apparatus embodiments provided below can be found in the limitations of the data processing method described above, and will not be repeated here.
[0135] In one embodiment, such as Figure 7 As shown, a data processing apparatus is provided, comprising: a first receiving module 701, a detection module 702, and a first storage module 703, wherein:
[0136] The first receiving module 701 is used to receive a data storage request sent by the second server. The data storage request includes the data to be stored and the first data primary key of the data to be stored.
[0137] The detection module 702 is used to detect, based on the first data primary key, whether there is a target data primary key that is the same as the first data primary key among the multiple second data primary keys included in the aggregated data table, and to obtain the detection result;
[0138] The first storage module 703 is used to store the data to be stored into the aggregated data table according to the detection results, and send a response message to the second server. The response message is used to indicate that the first server has stored the data to be stored.
[0139] The data processing device provided in this embodiment can execute the above method embodiment, and its implementation principle and technical effect are similar, so it will not be described again here.
[0140] In one embodiment, the first storage module 703 includes:
[0141] The first storage unit is used to perform aggregation processing on the data to be stored to obtain aggregated data if the detection result shows that the target data primary key exists among multiple second data primary keys, and then store the aggregated data in the aggregated data table.
[0142] The second storage unit is used to independently store the data to be stored in the aggregated data table if the detection result is that the target data primary key does not exist among the multiple second data primary keys.
[0143] The data processing device provided in this embodiment can execute the above method embodiment, and its implementation principle and technical effect are similar, so it will not be described again here.
[0144] In one embodiment, the first storage unit is specifically used for:
[0145] Retrieve the aggregation type corresponding to the primary key of the target data;
[0146] According to the aggregation rules corresponding to the aggregation type, the historical data corresponding to the primary key of the data to be stored and the target data are aggregated to obtain aggregated data.
[0147] Replace historical data stored in the aggregated data table with aggregated data.
[0148] The data processing device provided in this embodiment can execute the above method embodiment, and its implementation principle and technical effect are similar, so it will not be described again here.
[0149] In one embodiment, the data storage request further includes at least one first data tag, and the aforementioned apparatus further includes:
[0150] The second storage module is used to store the first data primary key and at least one first data tag in a tag index table in a reverse index manner. The tag index table includes multiple sets of correspondences, including the correspondence between the data primary key and at least one data tag.
[0151] The data processing device provided in this embodiment can execute the above method embodiment, and its implementation principle and technical effect are similar, so it will not be described again here.
[0152] In one embodiment, the above-mentioned apparatus includes:
[0153] The second receiving module is used to receive a data query request sent by the second server, the data query request including a second data tag;
[0154] The acquisition module is used to obtain the data to be queried corresponding to the data query request based on the second data label and the label index table.
[0155] The data processing device provided in this embodiment can execute the above method embodiment, and its implementation principle and technical effect are similar, so it will not be described again here.
[0156] In one embodiment, the above-mentioned acquisition module includes:
[0157] The first determining unit is configured to determine multiple target data labels from a label index table based on multiple second data labels, wherein the multiple target data labels are at least partially identical to the multiple second data labels.
[0158] The second determining unit uses the data primary key corresponding to each target data label to determine the data primary key set;
[0159] The retrieval unit is used to retrieve the data to be queried from the aggregated data table based on the data primary key set, and send the data to be queried to the second server.
[0160] The data processing device provided in this embodiment can execute the above method embodiment, and its implementation principle and technical effect are similar, so it will not be described again here.
[0161] Each module in the aforementioned data processing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.
[0162] In one exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 8As shown, this computer device includes a processor, memory, input / output interfaces (I / O), and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores aggregated data tables and tag index tables. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When the computer program is executed by the processor, it implements a data processing method.
[0163] Those skilled in the art will understand that Figure 8 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.
[0164] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:
[0165] Receive a data storage request sent by the second server. The data storage request includes the data to be stored and the first primary key of the data to be stored.
[0166] Based on the first primary data key, detect whether there is a target primary data key that is the same as the first primary data key among the multiple second primary data keys included in the aggregated data table, and obtain the detection result;
[0167] Based on the detection results, the data to be stored is stored in the aggregated data table, and a response message is sent to the second server. The response message is used to indicate to the first server that the data to be stored has been stored.
[0168] In one embodiment, the processor, when executing a computer program, also performs the following steps:
[0169] If the detection result shows that the target data primary key exists among multiple second data primary keys, then the data to be stored is aggregated to obtain aggregated data, and the aggregated data is stored in the aggregated data table;
[0170] If the detection result indicates that the target data primary key does not exist among multiple secondary data primary keys, the data to be stored will be stored independently in the aggregated data table.
[0171] In one embodiment, the processor, when executing a computer program, also performs the following steps:
[0172] Retrieve the aggregation type corresponding to the primary key of the target data;
[0173] According to the aggregation rules corresponding to the aggregation type, the historical data corresponding to the primary key of the data to be stored and the target data are aggregated to obtain aggregated data.
[0174] Replace historical data stored in the aggregated data table with aggregated data.
[0175] In one embodiment, the processor, when executing a computer program, also performs the following steps:
[0176] Using a reverse index, the first data primary key and at least one first data tag are stored in the tag index table. The tag index table includes multiple sets of correspondences, including the correspondence between the data primary key and at least one data tag.
[0177] In one embodiment, the processor, when executing a computer program, also performs the following steps:
[0178] Receive a data query request sent by the second server, the data query request including the second data tag;
[0179] Based on the second data label and the label index table, obtain the data to be queried corresponding to the data query request.
[0180] In one embodiment, the processor, when executing a computer program, also performs the following steps:
[0181] Multiple target data labels are determined from a label index table based on multiple second data labels, and the multiple target data labels are at least partially the same as the multiple second data labels;
[0182] The primary key corresponding to each target data label is determined as the set of primary keys;
[0183] The data to be queried is obtained from the aggregated data table based on the primary key set, and then sent to the second server.
[0184] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:
[0185] Receive a data storage request sent by the second server. The data storage request includes the data to be stored and the first primary key of the data to be stored.
[0186] Based on the first primary data key, detect whether there is a target primary data key that is the same as the first primary data key among the multiple second primary data keys included in the aggregated data table, and obtain the detection result;
[0187] Based on the detection results, the data to be stored is stored in the aggregated data table, and a response message is sent to the second server. The response message is used to indicate to the first server that the data to be stored has been stored.
[0188] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:
[0189] If the detection result shows that the target data primary key exists among multiple second data primary keys, then the data to be stored is aggregated to obtain aggregated data, and the aggregated data is stored in the aggregated data table;
[0190] If the detection result indicates that the target data primary key does not exist among multiple secondary data primary keys, the data to be stored will be stored independently in the aggregated data table.
[0191] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:
[0192] Retrieve the aggregation type corresponding to the primary key of the target data;
[0193] According to the aggregation rules corresponding to the aggregation type, the historical data corresponding to the primary key of the data to be stored and the target data are aggregated to obtain aggregated data.
[0194] Replace historical data stored in the aggregated data table with aggregated data.
[0195] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:
[0196] Using a reverse index, the first data primary key and at least one first data tag are stored in the tag index table. The tag index table includes multiple sets of correspondences, including the correspondence between the data primary key and at least one data tag.
[0197] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:
[0198] Receive a data query request sent by the second server, the data query request including the second data tag;
[0199] Based on the second data label and the label index table, obtain the data to be queried corresponding to the data query request.
[0200] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:
[0201] Multiple target data labels are determined from a label index table based on multiple second data labels, and the multiple target data labels are at least partially the same as the multiple second data labels;
[0202] The primary key corresponding to each target data label is determined as the set of primary keys;
[0203] The data to be queried is obtained from the aggregated data table based on the primary key set, and then sent to the second server.
[0204] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, performs the following steps:
[0205] Receive a data storage request sent by the second server. The data storage request includes the data to be stored and the first primary key of the data to be stored.
[0206] Based on the first primary data key, detect whether there is a target primary data key that is the same as the first primary data key among the multiple second primary data keys included in the aggregated data table, and obtain the detection result;
[0207] Based on the detection results, the data to be stored is stored in the aggregated data table, and a response message is sent to the second server. The response message is used to indicate to the first server that the data to be stored has been stored.
[0208] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:
[0209] If the detection result shows that the target data primary key exists among multiple second data primary keys, then the data to be stored is aggregated to obtain aggregated data, and the aggregated data is stored in the aggregated data table;
[0210] If the detection result indicates that the target data primary key does not exist among multiple secondary data primary keys, the data to be stored will be stored independently in the aggregated data table.
[0211] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:
[0212] Retrieve the aggregation type corresponding to the primary key of the target data;
[0213] According to the aggregation rules corresponding to the aggregation type, the historical data corresponding to the primary key of the data to be stored and the target data are aggregated to obtain aggregated data.
[0214] Replace historical data stored in the aggregated data table with aggregated data.
[0215] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:
[0216] Using a reverse index, the first data primary key and at least one first data tag are stored in the tag index table. The tag index table includes multiple sets of correspondences, including the correspondence between the data primary key and at least one data tag.
[0217] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:
[0218] Receive a data query request sent by the second server, the data query request including the second data tag;
[0219] Based on the second data label and the label index table, obtain the data to be queried corresponding to the data query request.
[0220] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:
[0221] Multiple target data labels are determined from a label index table based on multiple second data labels, and the multiple target data labels are at least partially the same as the multiple second data labels;
[0222] The primary key corresponding to each target data label is determined as the set of primary keys;
[0223] The data to be queried is obtained from the aggregated data table based on the primary key set, and then sent to the second server.
[0224] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.
[0225] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.
[0226] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0227] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.
Claims
1. A data processing method, characterized by, The method for a first server comprises: receiving a data storage request sent by a second server, the data storage request comprising to-be-stored data and a first data primary key of the to-be-stored data; detecting, according to the first data primary key, whether there is a target data primary key identical to the first data primary key in a plurality of second data primary keys comprised in an aggregated data table, to obtain a detection result; storing the to-be-stored data into the aggregated data table according to the detection result, and sending a response message to the second server, the response message being used to indicate that the first server has stored the to-be-stored data.
2. The method of claim 1, wherein, The storing of the to-be-stored data into the aggregated data table according to the detection result, and the sending of the response message to the second server, the response message being used to indicate that the first server has stored the to-be-stored data, comprises: if the detection result is that the target data primary key exists in the plurality of second data primary keys, performing an aggregation processing on the to-be-stored data to obtain aggregated data, and storing the aggregated data into the aggregated data table; if the detection result is that the target data primary key does not exist in the plurality of second data primary keys, storing the to-be-stored data into the aggregated data table independently.
3. The method of claim 2, wherein, The aggregation of the to-be-stored data to obtain aggregated data, and the storing of the aggregated data into the aggregated data table, comprises: obtaining an aggregation type corresponding to the target data primary key; performing an aggregation processing on the to-be-stored data and historical data corresponding to the target data primary key according to an aggregation rule corresponding to the aggregation type, to obtain the aggregated data; replacing the historical data stored in the aggregated data table with the aggregated data.
4. The method of claim 1, wherein, The data storage request further comprises at least one first data label, and the method further comprises: storing the first data primary key and the at least one first data label into a label index table in a reverse index manner, the label index table comprising a plurality of corresponding relationships, the corresponding relationship comprising a corresponding relationship between a data primary key and at least one data label.
5. The method of claim 4, wherein, The method further comprises: receiving a data query request sent by the second server, the data query request comprising a second data label; obtaining to-be-queried data corresponding to the data query request according to the second data label and the label index table.
6. The method of claim 5, wherein, The number of the second data labels is a plurality, and the obtaining of the to-be-queried data corresponding to the data query request according to the second data label and the label index table comprises: determining a plurality of target data labels from the label index table according to the plurality of second data labels, the plurality of target data labels being at least partially identical to the plurality of second data labels; determining data primary keys corresponding to each of the target data labels as a data primary key set; According to the data primary key set, the to-be-queried data is obtained from the aggregated data table, and the to-be-queried data is sent to the second server.
7. A data processing apparatus, characterized by, For a first server, the apparatus comprises: A first receiving module configured to receive a data storage request sent by a second server, the data storage request comprising to-be-stored data and a first data primary key of the to-be-stored data; A detecting module configured to detect, according to the first data primary key, whether a target data primary key identical to the first data primary key exists in a plurality of second data primary keys included in an aggregated data table, to obtain a detection result; A first storage module configured to store the to-be-stored data into the aggregated data table according to the detection result, and send a response message to the second server, the response message being used to indicate that the first server has stored the to-be-stored data.
8. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that, The processor executes the computer program to implement the steps of the method in any one of claims 1 to 6.
9. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method in any one of claims 1 to 6.
10. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method in any one of claims 1 to 6.