A data processing method, system and related devices

Abstract

The application provides a data processing method, a system and related equipment, relates to the technical field of computers, and the data processing method is applied to a data processing system, and the method comprises the following steps: a PDC determines a target weight parameter corresponding to each BDC according to a request frequency and a request type of each BDC, the target weight parameter is used for representing the probability that the BDC expects to store data of a first type, the request frequency represents the frequency at which the BDC requests data change from the PDC, and the request type represents the type of data requested by the BDC from the PDC; the PDC determines a first BDC from the multiple BDCs based on the target weight parameter corresponding to the type of the first data; and the PDC sends a first data change message to the first BDC, and the first data change message is used for changing the first data in the first BDC. The dynamic weighted distribution of the data storage task is realized, the BDC with a specific data type requirement is preferentially processed, the system resource distribution is optimized, the rapid growth of the data volume can be adapted to, and the data processing efficiency is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, and in particular, to a data processing method, system and related equipment. BACKGROUND

[0002] Data backup is a core means to resist data loss risk, guarantee business continuity and system recovery capability, which can effectively avoid irreparable loss caused by accidents, failures or attacks. At present, multi-point data backup usually needs to store data in multiple physical or virtual locations such as cloud, external hard disk, etc. By periodic data copying, data is synchronized to each backup node, thereby building a data security protection system and guaranteeing data recovery capability. However, this method relies on manual intervention and complex configuration, and is difficult to adapt to the rapid growth of data volume, resulting in low data processing efficiency. SUMMARY

[0003] Embodiments of the present application provide a data processing method, system and related equipment to solve the problem of low data processing efficiency in the prior art.

[0004] To solve the above technical problems, the present application is implemented as follows:

[0005] In a first aspect, the embodiments of the present application provide a data processing method applied to a data processing system, wherein the system comprises a center data processing center PDC and a plurality of backup data processing centers BDCs in communication connection with the PDC, and the plurality of BDCs are deployed at the edge of a communication network.

[0006] The method comprises:

[0007] The PDC determines a target weight parameter corresponding to each BDC according to a request frequency and a request type of each BDC, wherein the target weight parameter is used to represent a probability that the BDC expects to store data of a first type, the request frequency represents a frequency of the BDC requesting data change to the PDC, and the request type represents a type of data that the BDC requests data change to the PDC.

[0008] The PDC determines a first BDC from the plurality of BDCs based on the target weight parameter corresponding to the type of the first data.

[0009] The PDC sends a first data change message to the first BDC, wherein the first data change message is used to change the first data in the first BDC.

[0010] Optionally, the PDC determines a target weight parameter corresponding to each BDC according to a request frequency and a request type of each BDC, comprising:

[0011] The PDC determines a first weight parameter corresponding to each BDC according to a periodically acquired request frequency of each BDC, the first weight parameter being used to represent a probability that the BDC expects to store data;

[0012] The PDC determines a second weight parameter corresponding to each BDC according to a periodically acquired request type of each BDC, the second weight parameter being used to represent a probability that the BDC expects to store data associated with the corresponding request type;

[0013] The PDC obtains the target weight parameter according to the first weight parameter and the second weight parameter.

[0014] Optionally, the PDC determines a second weight parameter corresponding to each BDC according to a periodically acquired request type of each BDC, and the method comprises:

[0015] The PDC determines a first request proportion of the first type in the target BDC according to a first number and a second number, the first number being a number of times that the target BDC requests data change of the first type to the PDC, and the second number being a total number of times that the target BDC requests data change to the PDC, the target BDC being any one of the plurality of BDCs;

[0016] The PDC determines a second request proportion of the first type in the plurality of BDCs according to a third number and a fourth number, the third number being a sum of numbers of times that the plurality of BDCs requests data change of the first type to the PDC, and the fourth number being a sum of total numbers of times that the plurality of BDCs requests data change to the PDC;

[0017] The PDC determines a relative entropy of the target BDC according to the first request proportion and the second request proportion.

[0018] The PDC determines a second weight parameter of the target BDC according to the relative entropy of the target BDC.

[0019] Optionally, the PDC sends a first data change message to the first BDC, and the method comprises:

[0020] The PDC encapsulates the obtained first data change message into a message event, the message event being used for transmission in a message queue;

[0021] The PDC sends the message event to the first BDC based on a message queue corresponding to the first BDC, each BDC of the plurality of BDCs corresponding to a message queue.

[0022] Optionally, the method further comprises:

[0023] When the second BDC receives a data query request sent by a target user, judging a backup state of a second type of data in the second BDC, the second type being a data type indicated by the data query request for querying, the second BDC being a BDC closest to the target user among the plurality of BDCs;

[0024] When the backup state indicates that the second BDC does not save the second type of data, the second BDC generates a data backup request, the data backup request being used to request the PDC for the second type of data;

[0025] The PDC generates a second data change message according to the data backup request sent by the second BDC, the second data change message being used for backup of the second type of data;

[0026] The second BDC performs backup of the second type of data in the second BDC according to the second data change message sent by the PDC.

[0027] In a second aspect, an embodiment of the present application provides a data processing system, the system comprising a central data processing center PDC and a plurality of backup data processing centers BDCs in communication connection with the PDC, the plurality of BDCs being deployed at edges of a communication network; wherein,

[0028] The PDC is configured to perform the following operations:

[0029] According to a request frequency and a request type of each BDC, a target weight parameter corresponding to each BDC is determined, the target weight parameter being used to represent a probability that the BDC expects to store a first type of data, the request frequency representing a frequency at which the BDC requests data change from the PDC, and the request type representing a type of data for which the BDC requests data change from the PDC;

[0030] Based on the target weight parameter corresponding to the type of the first data, a first BDC is determined among the plurality of BDCs;

[0031] A first data change message is sent to the first BDC, the first data change message being used to perform change of the first data in the first BDC.

[0032] Optionally, the PDC is specifically configured to:

[0033] According to a periodically acquired request frequency of each BDC, a first weight parameter corresponding to each BDC is determined, the first weight parameter being used to represent a probability that the BDC expects to store data;

[0034] According to a request type of each BDC acquired periodically, a second weight parameter corresponding to each BDC is determined, the second weight parameter being used to represent a probability that the BDC expects to store data associated with the corresponding request type;

[0035] According to the first weight parameter and the second weight parameter, the target weight parameter is obtained.

[0036] Optionally, the PDC is specifically used for:

[0037] According to a first number and a second number, a first request proportion of the first type in a target BDC is determined, the first number being a number of times that the target BDC requests data change of the first type to the PDC, the second number being a total number of times that the target BDC requests data change to the PDC, the target BDC being any one of the plurality of BDCs;

[0038] According to a third number and a fourth number, a second request proportion of the first type in the plurality of BDCs is determined, the third number being a sum of numbers of times that the plurality of BDCs requests data change of the first type to the PDC, the fourth number being a sum of total numbers of times that the plurality of BDCs requests data change to the PDC;

[0039] According to the first request proportion and the second request proportion, a relative entropy of the target BDC is determined.

[0040] According to the relative entropy of the target BDC, a second weight parameter of the target BDC is determined.

[0041] In a third aspect, an electronic device is provided, which includes a processor, a memory, and a program stored in the memory and capable of running on the processor, and when the program is executed by the processor, the steps of the data processing method according to the first aspect are implemented.

[0042] In a fourth aspect, a computer readable storage medium is provided, and the computer readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the data processing method according to the first aspect are implemented.

[0043] In the embodiment of the present application, the data request frequency and request type of the BDC are analyzed by the PDC to determine the target weight parameter representing the data storage adaptation probability, and the target weight parameter is used to intelligently match the optimal BDC for a specific type of data to send a change message, thereby realizing dynamic weighted allocation of data storage tasks, and thus prioritizing the processing of the BDC with specific data type requirements, and optimizing system resource allocation. Compared with the manual intervention method, the intelligent message allocation ensures real-time and accurate synchronization of data changes, can adapt to rapid growth of data volume, and improves data processing efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0044] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed in the embodiments or prior art description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0045] Figure 1 is one of the flowcharts of the data processing method provided by the embodiments of the present application;

[0046] Figure 2 is a structural schematic diagram of a data processing system provided by the embodiments of the present application;

[0047] Figure 3 is the second flowchart of the data processing method provided by the embodiments of the present application. DETAILED DESCRIPTION

[0048] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0049] The terms "first", "second", and the like in the embodiments of the present application are used to distinguish similar objects, and do not necessarily mean a specific order or sequence. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not necessarily limit to those clearly listed steps or units, but can include other steps or units not clearly listed or inherent to the process, method, product or device.

[0050] Referring to Figure 1 and Figure 2 , Figure 1is one of flowcharts of a data processing method provided by an embodiment of the present application, Figure 2 is a structural schematic diagram of a data processing system provided by an embodiment of the present application. The data processing method provided by an embodiment of the present application can be applied to a data processing system. The data processing system can be a distributed system architecture composed of a primary data processing center (PDC) and a plurality of backup data processing centers (BDCs) in communication connection with the PDC. The plurality of BDCs are deployed at the edge of a communication network to reduce data transmission delay and improve response speed. The BDCs can also serve as edge service nodes for serving users in close physical proximity to the edge service nodes. These nodes are connected through a network and exchange data through a message queue.

[0051] The method comprises the following steps:

[0052] Step 101, the PDC determines a target weight parameter corresponding to each BDC according to a request frequency and a request type of each BDC. The target weight parameter is used to represent a probability that the BDC expects to store data of a first type. The request frequency represents a frequency of data change requested by the BDC from the PDC, and the request type represents a type of data requested by the BDC from the PDC.

[0053] In this step, the PDC dynamically calculates the target weight parameter corresponding to each BDC by analyzing the data request behavior of each BDC, in combination with the frequency of data change requested by each BDC from the PDC (i.e., the request frequency) and the specific type of data change requested (i.e., the request type). The target weight parameter quantifies the degree of inclination of the BDC to a specific type (e.g., the first type). The target weight parameter corresponding to the BDC is assigned a higher value, so that the priority and expected probability of the BDC in processing data of the first type are increased, thereby providing a data basis for subsequent intelligent allocation.

[0054] Step 102, the PDC determines a first BDC from the plurality of BDCs based on the target weight parameter corresponding to the type of the first data.

[0055] In this step, the first data can be data in the PDC that needs to be backed up to the BDC, and the type of the first data can be any one of the types of bills, subscription relationships, user home provinces, etc. The target weight parameter corresponding to the type of the first data can be different for each PDC. The higher the value of the target weight parameter, the more the BDC corresponding to the target weight parameter tends to process the first data. In other words, the higher the target weight parameter, the more likely the BDC is to be selected as the first BDC in the probabilistic sense. In this way, based on the target weight parameter corresponding to the type of the first data, the first BDC is determined among multiple BDCs, ensuring that the node with the type of the first data and the historical request characteristics of the first BDC are more matched to undertake the data change task with a higher probability, thereby optimizing the adaptability of data storage and the overall system efficiency.

[0056] In step 103, the PDC sends a first data change message to the first BDC, and the first data change message is used to change the first data in the first BDC.

[0057] In this step, after determining the first BDC that carries the data change, the PDC sends a first data change message containing complete change information to the first BDC. The message carries key instructions (such as add, delete, and modify operations, data mirroring, etc.) required to reproduce data changes at the BDC end, which is used to drive the first BDC to perform corresponding write operations based on the received change content, thereby realizing data synchronization between the PDC and the BDC, and ensuring that the backup data processing center can reflect the data state change of the center data processing center in real time and accurately.

[0058] In the embodiments of the present application, the data request frequency and request type of the BDC are dynamically analyzed by the PDC to determine the target weight parameter representing the data storage adaptation probability, and the target weight parameter is used to intelligently match the optimal BDC for a specific type of data for change message sending, thereby realizing dynamic weighted allocation of data storage tasks, prioritizing processing of BDCs with specific data type requirements, and optimizing system resource allocation. Compared with the manual intervention method, intelligent message allocation ensures real-time and accurate synchronization of data changes, can adapt to rapid growth of data volume, and improves data processing efficiency.

[0059] Optionally, the PDC determines the target weight parameter corresponding to each BDC according to the request frequency and the request type of each BDC, comprising:

[0060] The PDC determines a first weight parameter corresponding to each BDC according to the periodically obtained request frequency of each BDC, and the first weight parameter is used to represent the probability that the BDC expects to store data;

[0061] The PDC determines a second weight parameter corresponding to each BDC according to the request type of each BDC periodically acquired, and the second weight parameter is used to represent the probability that the BDC expects to store the data associated with the corresponding request type;

[0062] The PDC obtains the target weight parameter according to the first weight parameter and the second weight parameter.

[0063] In this embodiment, the PDC periodically acquires the data request frequency (i.e. the frequency of data change requests) and the request type (i.e. the specific requested data type) of each BDC, respectively generates a first weight parameter (based on frequency) representing the overall data storage adaptation probability of the BDC and a second weight parameter (based on type) representing the data storage adaptation probability for a specific request type, and finally forms a target weight parameter that comprehensively reflects the expected probability of the BDC for storing a specific type of data by fusing the weight parameters of the two dimensions, thereby providing a quantitative basis for dynamic intelligent allocation of data storage tasks.

[0064] Specifically, in the initial stage, a basic weight parameter is set for each BDC, and the basic weight parameters of all BDCs can be the same at the beginning (for example, if there are N BDCs, the basic weight parameter of each BDC is 1 / N). In addition, a global request counter is maintained for each data type to record the total number of requests for that type of data by all BDCs.

[0065] In the runtime phase, the request frequency is updated in real time, and a request frequency counter is maintained for each BDC and each data type. Each time a BDC requests data, the request frequency counter of the corresponding BDC and the corresponding data type is updated. In addition, by dividing the data types by geographical region or business type and analyzing their request frequencies, the data demand characteristics of different regions or business lines can be understood, providing strong support for formulating more accurate market strategies and business planning. According to the real-time updated request frequency, the layout and configuration of the BDC can be continuously optimized, the data processing efficiency and service quality can be improved, and the market competitiveness can be further enhanced.

[0066] In an example, the first weight parameter corresponding to each BDC can be determined as follows:

[0067] The PDC periodically (e.g. every hour or every day) calculates a new first weight parameter according to the request frequency of the BDC. The first weight parameter can be proportional to the request frequency, for example, using an exponential function or a logarithmic function of the request frequency as the basis for calculating the first weight parameter, to avoid weight skew in extreme cases. The formula for calculating the first weight parameter can be expressed as:

[0068] New first weight parameter = old first weight parameter * (1 + a * (request number - average request number) / average request number);

[0069] wherein the old first weight parameter is 1 / N in the initial state, a is an adjustment factor for controlling the sensitivity of weight adjustment, the request number represents the total number of times of data change requests from the BDC to the PDC, and the average request number represents the average value of the total number of times of data change requests from the BDC to the PDC.

[0070] In addition, to ensure that the sum of the first weight parameters corresponding to each BDC is 1, normalization processing can be performed so as to facilitate subsequent probability allocation.

[0071] Further, considering the data type tendency, the type (i.e., request type) of the request data of the BDC can be periodically analyzed to identify whether each BDC has a significant request type tendency. The PDC determines the second weight parameter corresponding to each BDC according to the request type of each BDC periodically acquired, and finally forms a target weight parameter that comprehensively reflects the expected probability of the BDC for a specific type of data storage by fusing the first weight parameter and the second weight parameter, thereby providing a quantitative basis for dynamic intelligent allocation of data storage tasks.

[0072] In some optional embodiments, the second weight parameter corresponding to each BDC can be determined by comparing the difference between the data type distribution requested by the BDC and the global data type distribution. For details, see the following description:

[0073] The PDC determines a first request proportion of the first type in the target BDC according to a first number and a second number, the first number being the number of times of data change requests of the first type from the target BDC to the PDC, and the second number being the total number of times of data change requests from the target BDC to the PDC, the target BDC being any one of the plurality of BDCs;

[0074] The PDC determines a second request proportion of the first type in the plurality of BDCs according to a third number and a fourth number, the third number being the sum of the number of times of data change requests of the first type from the plurality of BDCs to the PDC, and the fourth number being the sum of the total number of times of data change requests from the plurality of BDCs to the PDC;

[0075] The PDC determines the relative entropy of the target BDC according to the first request proportion and the second request proportion;

[0076] The PDC determines the second weight parameter of the target BDC according to the relative entropy of the target BDC.

[0077] In this embodiment, first, the use of each data type in the entire system is collected and counted, including the number of requests, data size, etc., to determine the global data type distribution. These data can come from the system logs, database query records or monitoring tools. In addition, the data type of each BDC request and its request number are collected, for example, through the logs or statistical information reported by the BDC when requesting data.

[0078] Then, the collected data is sorted to ensure that the statistical scope of each data type at the global and BDC levels is consistent; a unified naming rule is formulated for the data type to ensure the consistency and recognizability of the name; and the same time period is selected for comparison to eliminate errors caused by different time synchronization or data delay.

[0079] After that, according to the first quantity and the second quantity, the first request proportion of the first type in the target BDC is determined, the first quantity can be the number of times the target BDC requests data changes of the first type to the PDC, and the second quantity can be the total number of times the target BDC requests data changes to the PDC; in other words, the first request proportion can be the BDC data type proportion, for each BDC (taking the target BDC as an example), the proportion of each data type (taking the first type as an example) in its request can be calculated: the first request proportion=(the first quantity / the second quantity), that is, the proportion of the first type of data in the target BDC request=(the number of BDC requests of this data type / the total number of BDC requests).

[0080] According to the third quantity and the fourth quantity, the second request proportion of the first type in the plurality of BDCs is determined, the third quantity can be the sum of the number of times the plurality of BDCs requests data changes of the first type to the PDC, and the fourth quantity can be the sum of the total number of times the plurality of BDCs requests data changes to the PDC; in other words, the second request proportion can be the global data type proportion, for each data type (taking the first type as an example), the proportion of each data type in the global request is calculated: the second request proportion=(the third quantity / the fourth quantity), that is, the proportion of the first type of data in the global request=(the number of global requests of this data type / the total number of global requests).

[0081] After calculating the first request proportion and the second request proportion, according to the first request proportion and the second request proportion, the relative entropy of the target BDC is determined. The relative entropy is calculated, that is, the difference degree is calculated using the KL divergence (Kullback-Leibler Divergence). The KL divergence is a method for measuring the difference between two probability distributions P and Q, which can be used in this example to measure the difference between the BDC data type request distribution and the global data type distribution.

[0082] In an example, assume that the global data type distribution is P, where P(i) represents the request proportion of the i-th data type in the global (i.e., the second request proportion). Assume that the data type request distribution of a certain BDC is Q, where Q(i) represents the request proportion of the i-th data type in the BDC (i.e., the first request proportion). The KL divergence is calculated as follows:

[0083] ;

[0084] where D kl is the value of the KL divergence, and the smaller the value of the KL divergence, the closer the two distributions are; the larger the value, the greater the difference. For each BDC, the KL divergence between the BDC and the global data type distribution can be calculated to quantify the difference in data type requests.

[0085] The PDC determines the second weight parameter of each BDC according to the relative entropy of each BDC in the plurality of BDCs. In this way, the second weight parameter is set to achieve preferential allocation. For example, if it is found that a certain BDC has a significant tendency for a certain type of data, an additional second weight parameter for the type of data is added to the first weight parameter of the BDC. The size of the addition can be determined according to the degree of the tendency and the importance of the data type.

[0086] In this way, during each backup operation, the PDC randomly selects or proportionally allocates according to the current target weight parameter of the BDC. For example, a roulette wheel selection algorithm can be used to select the BDC according to the weight. Based on the target weight parameter corresponding to the type of the first data, the first BDC is determined among the plurality of BDCs, and the data type to which the first BDC has a tendency is preferentially sent to the first BDC; if there is no data type to which the first BDC has a tendency, the first BDC is randomly selected or proportionally allocated according to the global data type distribution. By combining the first weight parameter and the second weight parameter, the data type tendency of the BDC is considered, and the data to which the BDC has a tendency is preferentially sent to the BDC with a specific data type requirement. This targeted processing not only reduces unnecessary data transmission, but also shortens the response time of data requests and improves data processing efficiency. Moreover, based on the request frequency and data type tendency of the BDC, the target weight parameter of the BDC can be calculated and adjusted in real time. This dynamic weight adjustment mechanism ensures that the backup task can be flexibly allocated according to the actual demand, reducing resource waste or backup pressure concentration caused by static allocation, thereby further improving the overall backup efficiency.

[0087] Optionally, the PDC sends a first data change message to the first BDC, including:

[0088] The PDC encapsulates the obtained first data change message into a message event, and the message event is used for transmission in a message queue.

[0089] The PDC sends the message event to the first BDC based on the message queue corresponding to the first BDC, and each BDC in the plurality of BDCs corresponds to a message queue respectively.

[0090] In this embodiment, the PDC can change the associated database record data to a change log (such as binlog); a change log capture service (which can be a separate service or part of the database) monitors and captures new change log entries in real time. The PDC encapsulates the captured change log entries (i.e. the first data change message) into message events, each message event containing sufficient information to replay the corresponding data change on the BDC. Use a message queue (such as Kafka, RabbitMQ, etc.) as middleware for message events; the change log capture service publishes the encapsulated message events to the corresponding plurality of message queues (maintaining a message queue for each BDC) according to the message event distribution algorithm, waiting to be consumed; the data synchronization service on the BDC acts as a consumer of the message queue, subscribing to its own message queue; when new message events arrive, the data synchronization service asynchronously pulls these messages and processes them; for each message event, the data synchronization service parses the change instruction therein and performs the corresponding write operation on the database of the BDC to reproduce the data change on the PDC; after processing is complete, the data synchronization service sends an acknowledgement message to the message queue indicating that the message event has been successfully processed.

[0091] Among them, through the change log capture and encapsulation, the change log entries in the database are monitored and captured in real time, and are encapsulated into message events. This process realizes the instant capture of data changes, avoids the possible data omission in traditional backup, and thus ensures the integrity and accuracy of the backup data. At the same time, after encapsulation into message events, it is convenient for subsequent efficient processing and transmission, reduces the processing delay, and improves the backup efficiency.

[0092] Moreover, using a message queue as an intermediary, the encapsulated message events are distributed to the corresponding plurality of message queues according to the distribution algorithm. This way realizes the parallel processing of backup tasks, and multiple BDCs can receive and process backup data simultaneously, significantly improving the concurrency capability and processing speed of the backup operation.

[0093] In addition, the data synchronization service on the BDC asynchronously pulls and processes event messages, performs the corresponding write operation to reproduce the data change on the PDC. The asynchronous processing mechanism reduces the impact of the backup operation on the system main process, ensuring the high availability of the system. At the same time, the accurate change instruction reproduction ensures the consistency of the backup data with the original data, improving the accuracy of the backup.

[0094] For example, the PDC can be a MySQL database, and the BDC can be a MySQL database. Figure 3As shown, the data processing method provided in the embodiment can further include the following steps:

[0095] Step 301, PDC setting and initialization: the PDC sets and initializes the allocation algorithm of the message backup message, and can determine the target weight parameter through the allocation algorithm. The specific process can be referred to the above content, and will not be described here again. The algorithm is updated regularly to adapt to system changes.

[0096] Step 302, backup process: the PDC allocates the newly generated message to the corresponding BDC (edge service node) for backup according to the current message allocation algorithm.

[0097] Step 303, BDC saves messages: after obtaining the corresponding message, the BDC saves it in the local database.

[0098] Step 304, BDC implements user data request: the BDC simultaneously undertakes the functions of the edge router. When the user sends a data request to the corresponding edge router (i.e. the BDC), the BDC first searches whether it saves the requested data. If the BDC saves the requested data locally, it directly provides the data to the user. If the BDC does not save the requested data locally, the BDC will initiate a request to the PDC to provide the corresponding data. See the following description for details:

[0099] Optionally, the method further includes:

[0100] When the second BDC receives a data query request sent by a target user, the backup state of data of a second type in the second BDC is determined, the second type is a data type indicated by the data query request, and the second BDC is a BDC closest to the target user among the multiple BDCs;

[0101] When the backup state indicates that the second BDC does not save the data of the second type, the second BDC generates a data backup request, and the data backup request is used to request the data of the second type from the PDC;

[0102] The PDC generates a second data change message according to the data backup request sent by the second BDC, and the second data change message is used for backup of the data of the second type;

[0103] The second BDC performs backup of the data of the second type in the second BDC according to the second data change message sent by the PDC.

[0104] In this embodiment, the system first routes the user's request to the edge node (i.e., the second BDC) closest to the target user's physical location based on the geographical proximity principle to minimize network latency. Subsequently, the second BDC performs a quick check on its own storage state. It can determine whether the second type of data required by the request has been cached through the metadata index. If the check finds that the type of data is not stored, the second BDC immediately initiates an asynchronous data backup request to the PDC. This request can include information such as data type identification, timestamp, user geographical location, etc., to accurately describe the required data characteristics.

[0105] Upon receiving the request, the PDC invokes the data change generation module, which can construct a second data change message based on the global data view. This message contains incremental backup instructions, data verification mechanisms, and time point consistency guarantees. The generated change message can be transmitted to the second BDC through the message queue corresponding to the second BDC.

[0106] Upon receiving the change message, the second BDC starts the backup process. After the backup is complete, the second BDC returns the query result to the user and incorporates the newly backed up data into the local caching strategy, so that subsequent similar requests can be directly responded to at the edge node. In this way, the BDC not only serves as a storage center for message backup, but also integrates the function of an edge router, achieving localization and rapid response of data processing. This design simplifies the system architecture, reduces maintenance costs, and improves the processing speed of user requests, enabling users to obtain the required data more quickly.

[0107] The application also provides a data processing system, which includes a central data processing center PDC and a plurality of backup data processing centers BDCs in communication connection with the PDC, wherein the plurality of BDCs are deployed at the edge of a communication network.

[0108] The PDC is configured to perform the following operations:

[0109] Determine a target weight parameter corresponding to each BDC based on the request frequency and request type of each BDC, wherein the target weight parameter represents the probability that the BDC expects to store the first type of data, the request frequency represents the frequency of the BDC requesting data changes from the PDC, and the request type represents the type of data that the BDC requests data changes from the PDC;

[0110] Determine a first BDC from the plurality of BDCs based on the target weight parameter corresponding to the type of the first data;

[0111] Send a first data change message to the first BDC, wherein the first data change message is used to make changes to the first data at the first BDC.

[0112] Optionally, the PDC is specifically used for:

[0113] According to a request frequency of each BDC periodically acquired, a first weight parameter corresponding to each BDC is determined, the first weight parameter being used to represent a probability that the BDC expects to store data;

[0114] According to a request type of each BDC periodically acquired, a second weight parameter corresponding to each BDC is determined, the second weight parameter being used to represent a probability that the BDC expects to store data associated with the corresponding request type;

[0115] According to the first weight parameter and the second weight parameter, the target weight parameter is obtained.

[0116] Optionally, the PDC is specifically used for:

[0117] According to a first number and a second number, a first request proportion of a first type in a target BDC is determined, the first number being a number of times that the target BDC requests data change of the first type to the PDC, the second number being a total number of times that the target BDC requests data change to the PDC, the target BDC being any one of the plurality of BDCs;

[0118] According to a third number and a fourth number, a second request proportion of the first type in the plurality of BDCs is determined, the third number being a sum of numbers of times that the plurality of BDCs requests data change of the first type to the PDC, the fourth number being a sum of total numbers of times that the plurality of BDCs requests data change to the PDC;

[0119] According to the first request proportion and the second request proportion, a relative entropy of the target BDC is determined.

[0120] According to the relative entropy of the target BDC, a second weight parameter of the target BDC is determined.

[0121] Optionally, the PDC is specifically used for:

[0122] The obtained first data change message is encapsulated as a message event, the message event being used for transmission in a message queue;

[0123] Based on a message queue corresponding to the first BDC, the message event is sent to the first BDC, each BDC of the plurality of BDCs corresponding to a message queue respectively.

[0124] Optionally, a second BDC of the plurality of BDCs is used to perform the following operations:

[0125] When the second BDC receives a data query request sent by a target user, judging a backup state of a second type of data in the second BDC, the second type being a data type indicated by the data query request for querying, the second BDC being a BDC closest to the target user among the plurality of BDCs;

[0126] When the backup state indicates that the second type of data is not saved in the second BDC, the second BDC generates a data backup request for requesting the PDC for the second type of data;

[0127] The PDC is further configured to perform the following operation:

[0128] According to the data backup request sent by the second BDC, a second data change message is generated, the second data change message being used for backup of the second type of data;

[0129] The second BDC is further configured to perform the following operation:

[0130] According to the second data change message sent by the PDC, the second type of data is backed up in the second BDC.

[0131] The data processing system corresponds to each process of each embodiment of the data processing method in a one-to-one manner, and can achieve the same technical effects. To avoid repetition, details are not repeated here.

[0132] The embodiment of the present application also provides an electronic device, which comprises a processor, a memory, and a program stored in the memory and executable on the processor. When the program is executed by the processor, each process of the data processing method embodiment is implemented, and the same technical effects can be achieved. To avoid repetition, details are not repeated here.

[0133] The embodiment of the present application also provides a computer readable storage medium, which stores a computer program. When the computer program is executed by a processor, each process of the data processing method embodiment is implemented, and the same technical effects can be achieved. To avoid repetition, details are not repeated here. The computer readable storage medium includes a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0134] The embodiment of the present application also provides a computer program product, which comprises computer instructions. When the computer instructions are executed by a processor, each process of the data processing method embodiment is implemented, and the same technical effects can be achieved. To avoid repetition, details are not repeated here.

[0135] It should be noted that, as used in this document, the terms "comprises" or "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element proceeded by "comprises... a" does not, without more constraints, exclude the presence of additional identical elements in the process, method, article, or apparatus that comprises the element. Furthermore, it should be noted that the methods and apparatuses of the present embodiments are not limited to the order of the steps for carrying out the methods or to the exact sequence of steps as described. For example, the steps of the described methods can be performed in other sequences, or additional, omitted, or combined steps can be added to the described methods. Furthermore, features described with respect to certain examples can be combined in other examples.

[0136] Those skilled in the art can clearly understand that the above-mentioned embodiment method can be realized by means of software and necessary general hardware platform, of course, it can also be realized by hardware, but in many cases, the former is a better embodiment. Based on such understanding, the technical solutions of the present application can be embodied in the form of a software product, which is stored in a storage medium (such as a ROM / RAM, magnetic disk, or optical disk) and includes a plurality of instructions for causing a terminal (which can be a mobile phone, computer, server, air conditioner, or network device) to execute the methods described in the various embodiments of the present application.

[0137] The embodiments of the present application are described above with reference to the accompanying drawings, but the present application is not limited to the above-described specific embodiments, and the above-described specific embodiments are merely illustrative rather than limiting, and those of ordinary skill in the art can make many forms under the inspiration of the present application without departing from the scope of the present application and the scope of protection of the claims.

Claims

1. A data processing method, characterized by, The application is applied to a data processing system, the system comprising a central data processing center PDC and a plurality of backup data processing centers BDCs connected in communication with the PDC, the plurality of BDCs being deployed at the edge of a communication network; The method comprises: The PDC determines a target weight parameter corresponding to each BDC according to a request frequency and a request type of each BDC, the target weight parameter being used to represent a probability that the BDC expects to store data of a first type, the request frequency representing a frequency at which the BDC requests data change from the PDC, and the request type representing a type of data that the BDC requests data change from the PDC; The PDC determines a first BDC from the plurality of BDCs based on the target weight parameter corresponding to the type of the first data; The PDC sends a first data change message to the first BDC, the first data change message being used to make change to the first data at the first BDC; The PDC determines a target weight parameter corresponding to each BDC according to a request frequency and a request type of each BDC, comprising: The PDC determines a first weight parameter corresponding to each BDC according to a periodically acquired request frequency of each BDC, the first weight parameter being used to represent a probability that the BDC expects to store data; The PDC determines a second weight parameter corresponding to each BDC according to a periodically acquired request type of each BDC, the second weight parameter being used to represent a probability that the BDC expects to store data associated with the request type; The PDC obtains the target weight parameter according to the first weight parameter and the second weight parameter.

2. The method of claim 1, wherein, The PDC determines a second weight parameter corresponding to each BDC according to a periodically acquired request type of each BDC, comprising: The PDC determines a first request proportion of a first type in a target BDC according to a first number and a second number, the first number being a number of times that the target BDC requests data change of the first type from the PDC, the second number being a total number of times that the target BDC requests data change from the PDC, and the target BDC being any one of the plurality of BDCs; The PDC determines a second request proportion of the first type in the plurality of BDCs according to a third number and a fourth number, the third number being a sum of numbers of times that the plurality of BDCs requests data change of the first type from the PDC, and the fourth number being a sum of total numbers of times that the plurality of BDCs requests data change from the PDC; The PDC determines a relative entropy of the target BDC according to the first request proportion and the second request proportion; The PDC determines a second weight parameter of the target BDC according to the relative entropy of the target BDC.

3. The method of claim 1, wherein, The PDC sends a first data change message to the first BDC, comprising: The PDC encapsulates the obtained first data change message into a message event, the message event being used for transmission in a message queue; The PDC sends the message event to the first BDC based on a message queue corresponding to the first BDC, each of the plurality of BDCs corresponding to a message queue.

4. The method according to any one of claims 1 to 3, characterized in that, The method further comprises: In a case where the second BDC receives a data query request sent by a target user, judging a backup state of a second type of data in the second BDC, the second type being a data type indicated by the data query request, and the second BDC being a BDC closest to the target user in the plurality of BDCs; In a case where the backup state indicates that the second BDC does not save the second type of data, the second BDC generates a data backup request, the data backup request being used to request the second type of data from the PDC; The PDC generates a second data change message according to the data backup request sent by the second BDC, the second data change message being used for backup of the second type of data; The second BDC performs backup of the second type of data in the second BDC according to the second data change message sent by the PDC.

5. A data processing system, characterized by The system comprises a central data processing center PDC and a plurality of backup data processing centers BDCs in communication connection with the PDC, the plurality of BDCs being deployed at edges of a communication network; wherein, The PDC is configured to perform the following operations: determining a target weight parameter corresponding to each BDC according to a request frequency and a request type of each BDC, the target weight parameter being used to represent a probability that a BDC expects to store a first type of data, the request frequency representing a frequency at which a BDC requests data change from the PDC, and the request type representing a type of data for which a BDC requests data change from the PDC; determining a first BDC in the plurality of BDCs based on the target weight parameter corresponding to a type of first data; and sending a first data change message to the first BDC, the first data change message being used to perform change of the first data in the first BDC. The PDC is specifically configured to: determine a first weight parameter corresponding to each BDC according to a periodically acquired request frequency of each BDC, the first weight parameter being used to represent a probability that a BDC expects to store data; determine a second weight parameter corresponding to each BDC according to a periodically acquired request type of each BDC, the second weight parameter being used to represent a probability that a BDC expects to store data associated with the request type; obtain the target weight parameter according to the first weight parameter and the second weight parameter.

6. The system of claim 5, wherein, The PDC is specifically configured to: determine a first request proportion of a first type in a target BDC according to a first number and a second number, the first number being a number of times that the target BDC requests data change of the first type from the PDC, and the second number being a total number of times that the target BDC requests data change from the PDC, the target BDC being any one of the plurality of BDCs. determining a second request proportion of the first type among the plurality of BDCs according to a third quantity and a fourth quantity, the third quantity being a sum of times that the plurality of BDCs requests data changes of the first type from the PDC, and the fourth quantity being a sum of total times that the plurality of BDCs requests data changes from the PDC; determining a relative entropy of the target BDC according to the first request proportion and the second request proportion; determining a second weight parameter of the target BDC according to the relative entropy of the target BDC.

7. An electronic device, comprising: comprising: a processor, a memory, and a program stored on the memory and executable on the processor, the program, when executed by the processor, implements steps of the data processing method according to any one of claims 1 to 4.

8. A computer-readable storage medium, characterized in that, a computer program is stored on the computer readable storage medium, and the computer program, when executed by a processor, implements steps of the data processing method according to any one of claims 1 to 4.

Images