Data processing methods, devices, storage media and software products
By using a dynamic indexing mechanism in a distributed system, data is assigned to partitions based on core and density, and partition parameters are updated. This solves the clustering fragmentation problem in streaming clustering and improves the accuracy of data flowing into the same node and the clustering effect.
Patent Information
- Application Number
- CN202110749898.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-07-02
- Publication Date
- 2026-11-10
- Estimated Expiration
- 2041-07-02
AI Technical Summary
In distributed systems, during streaming clustering, the probability of data from the same community flowing into the same node is low, leading to cluster fragmentation and affecting the clustering effect.
A dynamic indexing mechanism is adopted to determine the partition to which data belongs by partitioning parameters such as core and density, and to update the dynamic index according to the data. The partitions are dynamically adjusted to improve the accuracy of data flowing into the same node, including partition merging, splitting and timeliness management.
This increases the probability of data from the same community flowing into the same node, reduces the impact of cluster fragmentation, and improves the effectiveness and adaptability of streaming clustering.
Smart Images

Figure CN115563097B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of data analysis, and more particularly to a data processing method, apparatus, storage medium, and program product. Background Technology
[0002] Clustering is a technique for statistical analysis of data. Traditional clustering algorithms are generally used to process static, offline data. By performing offline clustering analysis, static data is divided into different clusters, making data within the same cluster have similar attributes. In today's e-commerce, finance, and other fields, offline analysis is time-consuming and insufficient in terms of timeliness to support business objectives. Often, it is necessary to divide data into clusters within a short period of time to take appropriate business measures for further processing of different clusters. That is, facing a continuous stream of streaming data, clustering analysis needs to be completed in real time, for example, within seconds or minutes. This is called real-time clustering / streaming clustering.
[0003] To handle massive amounts of data while ensuring computing performance (high concurrency, low latency, and scalability), distributed systems are often used in e-commerce, risk control, and other fields to support business operations. Distributed systems can support high-performance reception of massive amounts of real-time data and perform real-time business calculations on such data. Therefore, to ensure performance targets are met, distributed systems can be relied upon for streaming clustering of massive amounts of data. In a distributed system, there may be multiple nodes with computing capabilities. Aggregating all data on a single node for clustering analysis violates the principles of distributed architecture, and the computing resources (CPU, memory, disk, etc.) of a single node are insufficient to support real-time clustering analysis of massive amounts of data. Therefore, streaming clustering in a distributed system faces the "clustering segmentation problem": data originally clustered in the same community (with similar attributes) flows to different nodes within the distributed system due to the distributed architecture. The inherent physical isolation prevents them from clustering into the same community, resulting in them being divided into different communities, leading to unsatisfactory streaming clustering results.
[0004] Therefore, how to increase the probability of data belonging to the same community flowing into the same node, reduce the adverse effects of the "clustering fragmentation problem", and improve the streaming clustering effect has become a research hotspot in this field. Summary of the Invention
[0005] In view of this, a data processing method, apparatus, storage medium and program product are proposed. The data processing method of the present application embodiment can increase the probability of data belonging to the same community flowing into the same node, reduce the adverse effects of the "clustering fragmentation problem" and improve the streaming clustering effect.
[0006] In a first aspect, embodiments of this application provide a data processing method applied to a distributed system, the distributed system including a flow guidance node and multiple processing nodes, wherein the flow guidance node stores a dynamic index, the dynamic index including multiple partitions, each partition corresponding to a processing node, the method including: the flow guidance node receiving data, selecting the partition to which the data belongs from the multiple partitions according to the parameters of each partition in the dynamic index; the flow guidance node outputting the data to the processing node corresponding to the selected partition, wherein data belonging to the same partition is clustered on the processing node; the flow guidance node updating the dynamic index according to the data belonging to the selected partition.
[0007] According to the data processing method of this application embodiment, by using the partition parameters in the dynamic index, the partition to which the data belongs can be selected from multiple partitions in the dynamic index. This ensures that when data is output to the processing node corresponding to the selected partition according to the correspondence between the partition and the processing node, there is already data belonging to the selected partition on that processing node, thereby enabling data belonging to the same partition to be distributed to the same node. After selecting the partition to which the data belongs, the dynamic index can be updated according to the data belonging to the selected partition. This further improves the accuracy of data distribution when subsequent received data is output to the processing node according to the updated dynamic index, avoiding the problem of uncorrectable errors in fixed partitioning mechanisms, increasing the probability of data belonging to the same community flowing into the same node, reducing the adverse effects of the "clustering fragmentation problem", and improving the streaming clustering effect and adaptability to different application scenarios.
[0008] According to the first aspect, in a first possible implementation of the data processing method, the parameters of each partition include a core and a density, wherein the core indicates the central location of the partition and the density indicates the degree of closeness between the data belonging to the partition.
[0009] In this way, partition parameters can be used to reflect relevant information about the data in the partition, ensuring the accuracy of selecting the partition to which the data belongs based on the partition parameters.
[0010] According to the first possible implementation of the first aspect, in the second possible implementation of the data processing method, the step of selecting the partition to which the data belongs from the plurality of partitions based on the parameters of each partition in the dynamic index includes: determining the proximity of the data to the core of each of the plurality of partitions; and selecting the partition whose core is closest to the data as the partition to which the data belongs.
[0011] In this way, the partition affiliation of received data can be determined. The closer the data is to the core of the partition, the more accurate the data's affiliation to that partition becomes. Furthermore, determining the proximity between the partition core and the data is relatively simple, reducing the complexity of selecting the partition to which the data belongs.
[0012] According to the first aspect, and any possible implementation of the first aspect above, in a third possible implementation of the data processing method, updating the dynamic index based on the data belonging to the selected partition includes: updating the core and density of the selected partition based on the data belonging to the selected partition.
[0013] In this way, partition parameters can be updated without changing the partitioning method, making dynamic index updates simpler and more convenient. After selecting a partition to which data belongs, the data belonging to that partition changes. The core and density of the selected partition are updated based on the changed data. This allows the selection of a new partition to belong to when new data is received subsequently, based on the updated core and density. In other words, the updated core and density are used to determine whether new data is highly similar to the selected partition, improving the accuracy of data partitioning.
[0014] According to the first aspect, and any one of the first and second possible implementations of the first aspect, in a fourth possible implementation of the data processing method, updating the dynamic index based on the data belonging to the selected partition includes: updating the density of the selected partition based on the data belonging to the selected partition; determining that the updated density is greater than a first threshold, splitting the selected partition into multiple partitions in the dynamic index; and updating the core and density of each partition based on the data belonging to each partition in the dynamic index.
[0015] By changing the partitioning method through partition splitting, the partition parameters are also changed, thus enabling the dynamic index to be updated. Since partition splitting is performed when the density does not meet the threshold condition, the resulting dynamic index with better partition density after splitting is updated, making the selection of the partition to which data belongs based on the updated dynamic index more accurate.
[0016] According to the fourth possible implementation of the first aspect, in the fifth possible implementation of the data processing method, updating the dynamic index based on the data belonging to the selected partitions further includes: determining that the total number of partitions in the dynamic index is greater than a second threshold, selecting multiple partitions in the dynamic index for merging; and updating the core and density of the merged partitions based on the data belonging to the merged partitions.
[0017] In this way, both the partitioning method and partitioning parameters change, enabling the dynamic index to be updated. Since partition merging is performed when the total number of partitions does not meet a threshold, the resulting dynamic index, updated through merging, has a more optimal number of partitions. The total number of partitions is related to the number of partitioning parameters in the dynamic index; a more optimal number of partitions results in lower time and computational costs for selecting the partition to which the data belongs based on the parameters in the dynamic index. This allows for high computational efficiency while maintaining sufficient accuracy when selecting the partition to which data belongs based on the updated dynamic index.
[0018] According to the fifth possible implementation of the first aspect, in the sixth possible implementation of the data processing method, merging multiple partitions in the dynamic index includes: determining the proximity of every two partitions in the dynamic index; and merging the two closest partitions into one partition.
[0019] In this way, partition merging can be completed, reducing the total number of partitions after merging, so as to get closer to the threshold requirement.
[0020] According to the first aspect, and any possible implementation of the first aspect above, in the seventh possible implementation of the data processing method, the method further includes: determining whether the data belonging to the partition of the dynamic index has timeliness; deleting the dynamic index when some or all of the data loses timeliness, or updating the dynamic index according to the data with timeliness.
[0021] In this way, the negative impact of old data on dynamic indexes can be eliminated, further improving the accuracy of selecting the partition to which data belongs based on the dynamic index.
[0022] Secondly, embodiments of this application provide a data processing apparatus applied in a distributed system, the distributed system including a flow guidance node and multiple processing nodes, wherein the flow guidance node stores a dynamic index, the dynamic index including multiple partitions, each partition corresponding to a processing node, and the apparatus includes:
[0023] A partition determination module is set on the traffic distribution guidance node to receive data and select the partition to which the data belongs from the multiple partitions according to the parameters of each partition in the dynamic index.
[0024] A data output module is set on the diversion guidance node and is used to output the data to the processing node corresponding to the selected partition, wherein data belonging to the same partition is used for clustering on the processing node.
[0025] An index update module, set on the routing guidance node, is used to update the dynamic index based on the data belonging to the selected partition.
[0026] According to the second aspect, in a first possible implementation of the data processing device, the parameters of each partition include a core and a density, the core indicating the center location of the partition and the density indicating the degree of closeness between the data belonging to the partition.
[0027] According to the first possible implementation of the second aspect, in the second possible implementation of the data processing device, the step of selecting the partition to which the data belongs from the plurality of partitions based on the parameters of each partition in the dynamic index includes: determining the proximity of the data to the core of each of the plurality of partitions; and selecting the partition whose core is closest to the data as the partition to which the data belongs.
[0028] According to the second aspect, and any possible implementation of the second aspect above, in a third possible implementation of the data processing device, updating the dynamic index based on the data belonging to the selected partition includes: updating the core and density of the selected partition based on the data belonging to the selected partition.
[0029] According to the second aspect, and any one of the first and second possible implementations of the second aspect, in a fourth possible implementation of the data processing device, updating the dynamic index based on the data belonging to the selected partition includes: updating the density of the selected partition based on the data belonging to the selected partition; determining that the updated density is greater than a first threshold, splitting the selected partition into multiple partitions in the dynamic index; and updating the core and density of each partition based on the data belonging to each partition in the dynamic index.
[0030] According to the fourth possible implementation of the second aspect, in the fifth possible implementation of the data processing device, updating the dynamic index based on the data belonging to the selected partitions further includes: determining that the total number of partitions in the dynamic index is greater than a second threshold, selecting multiple partitions in the dynamic index for merging; and updating the core and density of the merged partitions based on the data belonging to the merged partitions.
[0031] According to the fifth possible implementation of the second aspect, in the sixth possible implementation of the data processing device, merging multiple partitions in the dynamic index includes: determining the proximity of every two partitions in the dynamic index; and merging the two closest partitions into one partition.
[0032] According to the second aspect, and any possible implementation of the second aspect above, in a seventh possible implementation of the data processing apparatus, the apparatus further includes:
[0033] The timeliness determination module, set on the diversion guidance node, is used to determine whether the data belonging to the partition of the dynamic index has timeliness.
[0034] An index processing module, located on the diversion guidance node, is used to delete the dynamic index when some or all of the data loses its timeliness, or to update the dynamic index based on the data that has timeliness.
[0035] Thirdly, embodiments of this application provide a data processing apparatus, including: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to execute one or more of the data processing methods described in the first aspect or various possible implementations of the first aspect.
[0036] Fourthly, embodiments of this application provide a non-volatile computer-readable storage medium storing computer program instructions thereon, which, when executed by a processor, implement one or more of the data processing methods described in the first aspect or various possible implementations of the first aspect.
[0037] Fifthly, embodiments of this application provide a computer program product including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code. When the computer-readable code is run in an electronic device, the processor in the electronic device executes one or more of the data processing methods described in the first aspect or various possible implementations of the first aspect.
[0038] These and other aspects of this application will become more apparent in the description of the following embodiments(s). Attached Figure Description
[0039] The accompanying drawings, which are included in and form part of this specification, illustrate exemplary embodiments, features, and aspects of this application together with the specification and serve to explain the principles of this application.
[0040] Figure 1 A schematic diagram of a clustering method in the prior art is shown.
[0041] Figure 2 A schematic diagram of the clustering method of prior art 2 is shown.
[0042] Figure 3 A schematic diagram of the clustering method of prior art 3 is shown.
[0043] Figure 4 An exemplary application scenario according to an embodiment of this application is shown.
[0044] Figure 5 An example of a data processing method according to an embodiment of this application is shown.
[0045] Figure 6 An example of a dynamic index according to an embodiment of this application is shown.
[0046] Figure 7 An example illustrating the degree of association between data and various partitions in a dynamic index according to an embodiment of this application is provided.
[0047] Figure 8 An example is shown of updating the core and density of the selected partition according to an embodiment of this application.
[0048] Figure 9 An example of a partitioning method according to an embodiment of this application is shown.
[0049] Figure 10 An example of the Euclidean distance between the cores of every two partitions after partition splitting according to an embodiment of this application is shown.
[0050] Figure 11 An example of partition merging according to an embodiment of this application is shown.
[0051] Figure 12 An example of a data processing method according to an embodiment of this application is shown.
[0052] Figure 13 An example of a time window according to an embodiment of this application is shown.
[0053] Figure 14 An exemplary flowchart of a data processing method according to an embodiment of this application is shown.
[0054] Figure 15 This diagram illustrates an exemplary structure of a data processing apparatus according to an embodiment of the present application.
[0055] Figure 16 This diagram illustrates an exemplary structure of a data processing apparatus according to an embodiment of the present application. Detailed Implementation
[0056] Various exemplary embodiments, features, and aspects of this application will now be described in detail with reference to the accompanying drawings. The same reference numerals in the drawings denote elements that have the same or similar functions. Although various aspects of the embodiments are shown in the drawings, they are not necessarily drawn to scale unless specifically indicated otherwise.
[0057] The term “exemplary” as used herein means “serving as an example, embodiment, or illustration.” Any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments.
[0058] Furthermore, to better illustrate this application, numerous specific details are provided in the following detailed embodiments. Those skilled in the art should understand that this application can be implemented without certain specific details. In some instances, methods, means, components, and circuits well-known to those skilled in the art have not been described in detail in order to highlight the main points of this application.
[0059] The following are definitions of the terms used in this article.
[0060] Clustering, also known as cluster analysis, is a technique for analyzing statistical data, widely used in machine learning, data mining, pattern recognition, image analysis, and bioinformatics. Clustering groups objects with similar attributes into different groups or subsets using static classification methods. This ensures that objects within the same group / subset share similar attributes, such as shorter spatial distances in a coordinate system. Each group or subset can be considered a community. Clustering is generally categorized as an unsupervised learning method where data is unlabeled, and in engineering, it typically involves offline batch statistical analysis of data.
[0061] Streaming clustering: In order to improve the efficiency of extracting and mining knowledge from data and ensure the timeliness of data knowledge, clustering methods are applied to real-time streaming data. Faced with a continuous stream of data, clustering analysis can be completed instantly, within seconds or minutes.
[0062] The following introduces several existing solutions to the "clustering fragmentation problem".
[0063] Figure 1 A schematic diagram of a prior art clustering method is shown. For example... Figure 1As shown, streaming data can include a user group whose users have similar attributes. According to existing technology, streaming data flows randomly to different nodes in a distributed system, and local clustering is performed on different nodes. After all nodes are clustered, the data (one or more users) on each node can be considered to correspond to one or more communities. On this basis, a "secondary global aggregation" is performed, that is, similar communities between nodes are clustered to obtain a community that includes the user group, thus avoiding the user group being physically isolated by different nodes.
[0064] The drawback of existing technology one is that "secondary global aggregation" requires waiting for the last distributed node to complete its local clustering. This waiting time depends on the clustering time of the node that last completed its local clustering, creating a bottleneck effect in terms of time performance and reducing the real-time performance of streaming clustering. Furthermore, the amount of data processed by "secondary global aggregation" is very large, significantly increasing computation time and reducing the computational efficiency of streaming clustering. In addition, "secondary global aggregation" requires complex application techniques, needing to perform secondary clustering based on the community features after local clustering of nodes, thus increasing the application complexity of streaming clustering.
[0065] Figure 2 A schematic diagram of the clustering method in prior art two is shown. For example... Figure 2 As shown, streaming data can include a user group whose users have similar attributes. Existing technology 2 proposes that before streaming data flows to different nodes in a distributed system, it first performs "pre-data partitioning" and adopts partitioning strategies such as locality sensitive hashing (LSH) and hashing algorithms to make the user group belong to the same partition as much as possible. Data belonging to the same partition flows to the same node, and data on different nodes are clustered locally to reduce the probability that data of the same community is physically isolated by different nodes.
[0066] The drawback of the existing technology 2 is that the partitioning mechanism of strategies such as locality-sensitive hashing and hashing algorithms is fixed. That is, if the partitioning result of a user using this mechanism is incorrect, making it impossible for that user to belong to the same partition as other users in the user group, and thus unable to flow to the same node as other users, then the partitioning result of another user with the same attributes will still be incorrect the next time. Therefore, the effect of clustering depends entirely on the fixed partitioning mechanism, making the probability that the user group is physically isolated by different nodes still relatively high.
[0067] Figure 3 A schematic diagram of the clustering method in prior art three is shown. For example... Figure 3As shown, the third prior art combines the solutions of the first and second prior art, but still fails to overcome the shortcomings of the first and second prior art.
[0068] To address the aforementioned technical problems, this application provides a data processing method, apparatus, storage medium, and program product. The data processing method in this application can increase the probability of data belonging to the same community flowing into the same node, reduce the adverse effects of the "clustering fragmentation problem," and improve the streaming clustering effect.
[0069] The data processing method described in this application can be applied to distributed systems. Figure 4 This illustration demonstrates an exemplary application scenario according to an embodiment of this application. In this scenario, a receiving device and multiple cluster nodes (the 1st to Nth cluster nodes) are configured in a distributed system. The receiving device can act as the master device of the distributed system, and the multiple cluster nodes can act as multiple sub-devices of the distributed system. The receiving device, sending device, and cluster nodes can be any type of device, including but not limited to smartphones, personal computers, tablets, etc. The receiving device can store a dynamic index, which may include several index partitions. The dynamic index can indicate different index partitions and the attribute information corresponding to each index partition. The dynamic index can be used to determine the index partition to which the received data belongs, and to cluster data belonging to the same index partition into the same sub-device. The dynamic index can be updated as the received data is received to increase the probability that data belonging to the same community flows into the same node. The creation and updating methods of the dynamic index are described below.
[0070] Table 1 shows an example of a data structure for a dynamic index according to an embodiment of this application.
[0071] Table 1
[0072] property M1; D1 M2; D2 … MQ; DQ partition 1 2 … Q
[0073] As shown in Table 1, a dynamic index includes index partitions and attributes. One index partition corresponds to only one cluster node, while one cluster node can correspond to multiple partitions. For example, when there are N cluster nodes (N≥1 and are integers) and the maximum number of index partitions is W (W≥N and are integers), partition 1 can represent the first index partition, and can be set to correspond to the first cluster node, for example; partition 2 can represent the second index partition, and can be set to correspond to the second cluster node, for example; partition Q can represent the Qth index partition, where N≤Q≤W and are integers, and can be set to correspond to the Nth cluster node, for example.
[0074] The attributes may include, for example, the core M and the density D. The core M describes the "central position" of the index partition, and the density D describes the degree of clustering or compactness of all data within the index partition. For example, if the data belonging to a partition is shaped approximately like a two-dimensional circle in a coordinate system, the center of the circle may indicate the core M, and the radius may indicate the density D. Under this definition, the density value is negatively correlated with the compactness. For example, with a constant amount of data, a smaller radius (smaller density value) indicates a greater degree of clustering of all data within the index partition, resulting in higher data compactness and stronger correlation between the data (e.g., closer proximity). Conversely, a larger radius (larger density value) indicates a lesser degree of clustering of all data within the index partition, resulting in lower data compactness and weaker correlation between the data (e.g., greater distance). Each index partition includes a set of core and density combinations, such as the combinations M1 and D1, M2 and D2, and MQ and DQ shown in Table 1.
[0075] exist Figure 4 In application scenarios, for an index partition consisting of several data entries, one possible definition of core M is the average value obtained by calculating each dimension of each data entry, and one possible definition of density D is the average distance of each data entry to core M.
[0076] Specifically, suppose partition 1 includes two data points, each with two dimensions a and b. In a two-dimensional coordinate system with dimensions a and b as axes, the position of the first data point can be, for example, (a1, b1), and the position of the second data point can be, for example, (a2, b2). Then, the average value of dimension a for each data point is A = (a1 + a2) / 2, and the average value of dimension b for each data point is B = (b1 + b2) / 2. The core M1 of partition 1 can be, for example, M1 = (A, B). The distance from the first data point to the core M1 can be, for example,... The distance from the second data point to core M1 can be, for example, as follows: The density D1 of partition 1 can be, for example, (d1+d2) / 2. Under this definition, the density value of a partition is positively correlated with the sum of the distances of all data belonging to the partition, and negatively correlated with the degree of compactness of the partition. That is, the smaller the density value, the more tightly the data within the partition is packed.
[0077] Those skilled in the art should understand that the above-mentioned core and density are only examples of partition attributes in dynamic indexes. Core M, density D, distance between data and index partitions, etc., can all be specifically quantified and expressed in other ways. For example, core M can also be determined based on the weighted average of the positions of each data item, etc. This application does not limit this.
[0078] like Figure 4 As shown, in an exemplary application scenario, the sending device can send streaming data to the receiving device. The receiving device can receive the streaming data. Upon receiving each piece of data, it can calculate the correlation between the data and different partitions of the dynamic index, and output the data to the cluster node corresponding to the partition with the strongest correlation. The method for calculating the correlation can be found below. Figure 7 And related descriptions.
[0079] Once the partition with the strongest correlation to the data is identified, the data entry can be assigned to that partition. As the data in the partition changes, the dynamic index is updated accordingly. The update method for the dynamic index is detailed below. Figure 8 , 9 The receiving device can use the latest dynamic index to determine the partition most strongly associated with the next data item. This allows the next data item to be output to the corresponding cluster node. Similarly, each data item in the streaming data can be output to a cluster node corresponding to its most strongly associated partition.
[0080] On each cluster node, streaming data is clustered locally. If a cluster node corresponds to multiple partitions, then data belonging to different partitions on that node are clustered locally separately. The clustering algorithm used by each node for local clustering can, for example, be the same clustering algorithm used to calculate the clustering degree when generating index partitions in a dynamic index.
[0081] The receiving device and cluster nodes can be different computing units located on the same computer; that is, a single computer can include the receiving device and / or at least one cluster node. When the receiving device itself has sufficient computing power, it can also function as a cluster node. This application does not restrict the configuration of the receiving device and cluster nodes.
[0082] The exemplary workflow of the data processing method according to the embodiments of this application will be described below with reference to the above exemplary application scenarios. The workflow of the data processing method may differ depending on whether a dynamic index exists on the receiving device. Figures 5-11 This document presents an example of a data processing method according to embodiments of this application when a dynamic index exists on the receiving device.
[0083] Figure 5 An example of a data processing method according to an embodiment of this application is shown. For example... Figure 5 As shown, in one possible implementation, the data processing method of this application embodiment is executed by a receiving device, including:
[0084] S21, Receive data from the transmitting device.
[0085] For example, the data could be a single data point in a streaming dataset, which could be video, audio, images, text, or other similar data. A single data point received could be one or more video frames, one or more audio frames, a single image, or a set of images. The sending device could, for example, transmit streaming data to the receiving device.
[0086] S22, determine whether a dynamic index already exists on the receiving device.
[0087] S23, if a dynamic index exists, select the partition to which the data belongs based on the dynamic index.
[0088] For example, the data received in step S21 could be the Xth data received after the start of receiving streaming data. Prior to this, the receiving device has already obtained a dynamic index based on data from data 1 to data (an exemplary method for obtaining a dynamic index based on data from data 1 to data (see...) Figure 12 , Figure 5 (and related explanations), the dynamic index may include, for example, four partitions. Figure 6 An example of a dynamic index according to an embodiment of this application is shown.
[0089] For example, the partition to which data belongs can be the partition with the strongest correlation to the data. The degree of correlation between data and partition can be determined through clustering calculation. The algorithm for clustering calculation can, for example, be the same algorithm used when performing local clustering on each cluster node. In this way, data that should be clustered into the same community is likely to be assigned to the same partition, thus entering the same cluster node, avoiding the "clustering fragmentation problem" where data that should be clustered into the same community ends up on different cluster nodes.
[0090] In one possible implementation, the attributes of each partition in the dynamic index include, for example, core and density. Aggregation calculation can determine the proximity of the data received in step S21 (the Xth data point) to the core of each partition. Proximity can be a quantitative expression of "association." For example, determining the Euclidean distance between the position of the Xth data point and the core in a two-dimensional coordinate system, or determining the Doppler distance between the Xth data point and the core, etc. Aggregation calculation can, for example, yield multiple distance calculation results, the number of which equals the number of partitions in the dynamic index. The numerical value of the result indicates the degree of association between the Xth data point and each partition in the dynamic index. That is, the larger the value, the greater the distance between the position of the Xth data point and the core of the current partition, and the weaker the association between the Xth data point and the current partition; the smaller the value, the smaller the distance between the position of the Xth data point and the core of the current partition, and the stronger the association between the Xth data point and the current partition.
[0091] Figure 7This illustrates an example of the degree of association between data and various partitions in a dynamic index according to an embodiment of this application. Figure 7 In the example, among multiple distance calculation results, the smallest value is obtained when the location of the Xth data point is aggregated with the core M1 of partition 1. This indicates that the distance between the location of the Xth data point and the core M1 of partition 1 is the smallest, and therefore the correlation between the Xth data point and partition 1 is the strongest. In this case, partition 1 can be the partition to which the selected data point belongs. If there are multiple smallest value calculation results, then any one of the multiple partitions corresponding to the cores used in the aggregation calculation when these multiple calculation results are obtained can be selected as the partition to which the Xth data point belongs.
[0092] S24. Based on the correspondence between the selected partition and the cluster node, the data is output to the cluster node corresponding to the selected partition.
[0093] For example, partition 1 may correspond to the first cluster node, partition 2 may correspond to the second cluster node, partition 3 may correspond to the third cluster node, and partition 4 may also correspond to the third cluster node. When the selected partition to which the data belongs is partition 1, the receiving device can output the data received in step S21 to the first cluster node.
[0094] S25, update the partition attributes in the dynamic index based on the data, or update the partitioning method and partition attributes in the dynamic index.
[0095] In one possible implementation, after selecting the partition to which the data belongs, in step S25, the core and density of the selected partition in the dynamic index can be updated.
[0096] Figure 8 This illustrates an example of updating the core and density of a selected partition according to an embodiment of this application. Figure 8 Taking an application scenario as an example, the data belonging to multiple partitions of a dynamic index can be, for instance, two-dimensional data with dimensions a and b. Before the arrival of the Xth data, there may already be several data (e.g., F data) belonging to partition 1. The core M1 and density D1 of this partition are determined based on the F data. After selecting the partition (partition 1) to which the Xth data belongs, the number of data belonging to partition 1 increases to F+1. In this case, the core M1 and density D1 of partition 1 may change. Based on the F+1 data, the core M1' and density D1' of partition 1 can be recalculated. The specific calculation method can be found in the above text. Figure 4 The corresponding description.
[0097] After the core M1' and density D1' are calculated, the dynamic index can be updated based on the core M1' and density D1'. The updated dynamic index has the same number of partitions as the original dynamic index. In the updated dynamic index, the core and density corresponding to partition 1 are the core M1' and density D1' calculated after assigning the Xth data to partition 1. Except for partition 1, the data assigned to other partitions (partitions 2, 3, and 4) remain unchanged. Therefore, the core and density of other partitions are the same as the core and density of the corresponding partitions in the original dynamic index.
[0098] The selected partition to which the data belongs is already the most suitable partition among the existing partitions. However, if the data differs significantly from the other data in the selected partition, for example, if it is far away from other data in a two-dimensional coordinate system, the data density of the partition may decrease. This would result in the selected partition having a higher density value and lower density. When the density of the partition is high, the accuracy of data distribution using dynamic indexes may be affected.
[0099] This application further proposes setting threshold conditions for attributes, and after selecting the partition to which the data belongs, judging the attributes of the partition, and further processing the dynamic index when the attributes do not meet the threshold conditions, so as to ensure the accuracy of data distribution by means of dynamic index.
[0100] In one possible implementation, after selecting the partition to which the data belongs, the density of the selected partition in the dynamic index can be updated in step S25.
[0101] The method for updating the density of the selected partitions can be found in [reference]. Figure 8 In the example, the data belonging to multiple partitions of the dynamic index can be, for example, two-dimensional data with dimensions a and b. Before the arrival of the Xth data, there may already be several data (e.g., F data) belonging to partition 1. The core M1 and density D1 of this partition are determined based on the F data. After selecting the partition (partition 1) to which the Xth data belongs, the number of data belonging to partition 1 increases to F+1. In this case, the core M1 and density D1 of partition 1 may change. Based on the F+1 data, the density D1' of partition 1 can be recalculated. The specific calculation method can be found in the above text. Figure 4 The corresponding description.
[0102] In one possible implementation, after the density D1' is calculated, step S25 further includes judging the value of the density, based on... Figure 4The definition of density in the relevant description states that the smaller the density value, the denser the data in the partition. When the calculated density is less than or equal to the first threshold, it is considered that the data distribution based on the current dynamic index is highly accurate, and there is no need to change the partitioning method of the dynamic index. In this case, the core of the partition to which the selected data belongs can be further updated to complete the update of the dynamic index.
[0103] For example, the first threshold U can be preset. When the density is less than or equal to the first threshold U, it can be considered that after assigning the Xth data to partition 1, all data belonging to that partition are sufficiently dense, and the correlation between the data is strong enough. In this case, assigning the Xth data to partition 1 is considered feasible, and the current dynamic index has high accuracy. After selecting the partition to which the data belongs, the core M1' of that partition can be calculated based on the data belonging to the selected partition. The calculation method can be referred to... Figure 8 Examples and Figure 4 The relevant description is as follows. The density D1' of the selected partition has been calculated. Therefore, after the core M1' is calculated, the dynamic index can be updated based on the core M1' and density D1'. The updated dynamic index has the same number of partitions as the original dynamic index. In the updated dynamic index, the core and density corresponding to partition 1 are the core M1' and density D1' calculated after assigning the Xth data to partition 1. Except for partition 1, the data assigned to other partitions (partitions 2, 3, 4) remain unchanged. Therefore, the core and density of other partitions are the same as the core and density of the corresponding partitions in the original dynamic index.
[0104] In one possible implementation, after determining the density value and confirming that the density is greater than a first threshold U, the current dynamic index is considered to have low accuracy. Step S25 also includes steps S31-S32 described below to process the dynamic index and update the partitioning method and partitioning attributes in the dynamic index.
[0105] S31, the selected partition in the dynamic index is split into multiple partitions.
[0106] Taking the split into two partitions as an example, Figure 9 An example of a partitioning method according to an embodiment of this application is shown. For example... Figure 9As shown, when partition 1 is selected, two data items can be randomly selected as possible partition cores based on the F data items and the Xth data item within the selected partition (partition 1). The two partition cores are first assigned to two partitions (e.g., partition 1-1 and partition 1-2), and then the remaining data is assigned to the two partitions. As the number of data items in the two partitions increases, the partition cores also change. This continues until all F data items and the Xth data item are assigned. Partition splitting can be implemented using existing technology. This application does not limit the partition splitting method; for example, other partition splitting methods can be used to split into two partitions, or split into two or more partitions.
[0107] S32 calculates the core and density of each partition after splitting, and updates the dynamic index.
[0108] The calculation methods for core and density can be found in the above text. Figure 4 The relevant description is as follows: After the core and density are calculated, the dynamic index can be updated based on the core and density. Taking the selected partition (partition 1) split into two partitions (partition 1-1 and partition 1-2) as an example, the number of partitions in the updated dynamic index is one more than the number of partitions in the original dynamic index. In this case, partition 1-1 obtained after splitting partition 1 in the original dynamic index can be used as partition 1 in the updated dynamic index; partitions 2, 3, and 4 in the original dynamic index can be used as partitions 2, 3, and 4 in the updated dynamic index, respectively; partition 1-2 obtained after splitting partition 1 in the original dynamic index can be used as partition 5 in the updated dynamic index. The core and density corresponding to partition 1 are the calculated core and density of partition 1-1, and the core and density corresponding to partition 5 are the calculated core and density of partition 1-2. Except for partitions 1 and 5, the data belonging to other partitions (partitions 2, 3, and 4) remains unchanged. Therefore, the core and density of other partitions are the same as the core and density of the corresponding partitions in the original dynamic index.
[0109] Partition 5 is a new partition in the dynamic index. The correspondence between partition 5 and cluster nodes can be pre-set. For example, it can be pre-set that the 5th partition appearing in the dynamic index in chronological order (i.e., partition 5) corresponds to a specific cluster node, such as the 3rd cluster node. Alternatively, the correspondence between partition 5 and cluster nodes can be set to preferentially select cluster nodes with fewer corresponding partitions. For example, if the 3rd cluster node does not yet have a corresponding partition, then the 3rd cluster node can be preferentially selected as the cluster node corresponding to partition 5. Those skilled in the art should understand that there are multiple ways to set the correspondence between partitions and cluster nodes, and this application does not limit this.
[0110] After the receiving device selects the partition to which a piece of data belongs based on a given data entry, the data is distributed to the cluster node corresponding to that partition. On the cluster node, data from the same partition undergoes a local clustering process. In other words, the number of local clustering calculations is related to the total number of partitions. If the number of partitions is too large, the number of clustering operations on the cluster node will increase significantly, potentially exceeding the computing capacity of the cluster node and affecting the clustering results.
[0111] This application further proposes setting a threshold condition for the number of partitions, judging the total number of partitions after partition splitting, and further processing the dynamic index when the total number of partitions does not meet the threshold condition requirement, so as to reduce the number of partitions and ensure the clustering effect.
[0112] In one possible implementation, after executing steps S31-32 to complete the partition splitting and dynamic index update, step S25 further includes judging the number of partitions after the partition splitting. If the total number of partitions after splitting does not exceed the partition number threshold, it is considered that after the data is distributed according to the current dynamic index partitioning method, the cluster nodes can achieve good clustering, and no further processing of the dynamic index is required.
[0113] For example, the partition number threshold W can be preset based on the computing power of the cluster nodes. For instance, if W = 5, before partition splitting is completed in step S31, the number of partitions can be, for example, 4. After partition splitting, the total number of partitions increases by one (to 5). In this case, the number of partitions is less than or equal to the partition number threshold W, so the partitioning method after splitting can be considered feasible, and the current dynamic index partitioning method meets the requirements. Furthermore, under the current dynamic index partitioning method, the dynamic index has already been updated (step S32). Therefore, no further processing of the current dynamic index is required before receiving the next piece of data.
[0114] In one possible implementation, after determining the total number of partitions after partition splitting and confirming that the total number of partitions is greater than the partition number threshold W (for example, when the partition number threshold W is set to 4), the total number of partitions before partition splitting in step S31 can be, for example, 4. After partition splitting, the total number of partitions increases by one (5). In this case, the total number of partitions in the dynamic index is greater than the partition number threshold W, and the current partitioning method can be considered infeasible, and the current dynamic index is inefficient. Step S25 also includes steps S33-S35 below to process the dynamic index and update the partitioning method and partition attributes in the dynamic index.
[0115] S33, based on the core of all partitions in the dynamic index after the split, determine the proximity of every two partitions after the partition split.
[0116] Proximity can be measured by metrics such as Euclidean distance or Doppler distance between cores. It can also be measured by the similarity between two partitions, such as the difference in values along the same dimension or shared characteristics. Proximity calculations can be performed using existing technologies. Figure 10 An example of the Euclidean distance between the cores of every two partitions after partition splitting according to an embodiment of this application is shown.
[0117] Those skilled in the art should understand that the method of calculating the degree of proximity is not limited to this. As long as the degree of proximity between the two partitions can be determined by the relevant information of the two partitions, including but not limited to core attributes, and the data belonging to the two partitions, this application does not limit the method of determining the degree of proximity.
[0118] S34 merges the two closest partitions into one partition.
[0119] For example, the closest partitions can be those whose cores are closest in Euclidean distance. Figure 11 An example of partition merging according to an embodiment of this application is shown. Figure 11 As shown, taking a split into two partitions as an example, the dynamic index before the split includes partitions 1, 2, 3, and 4, and the dynamic index after the split includes partitions 1, 2, 3, 4, and 5. Calculations show that the Euclidean distance between the cores of partitions 2 and 3 is the smallest, meaning partitions 2 and 3 are closest. In this case, partitions 2 and 3 can be merged. For example, after merging the original partitions 2 and 3, they become partition 2. The original partitions 1 and 4 remain as partitions 1 and 4, and the original partition 5 becomes partition 3. In this case, the total number of partitions equals the partition count threshold, and the merged partitioning method can be considered feasible.
[0120] S35 calculates the core and compactness of the merged partition and updates the dynamic index.
[0121] For example, after merging, the core and density of the merged partitions can be calculated. The calculated core and density can then be used to update the attributes of the corresponding partition (partition 2) in the dynamic index. The attributes of partitions 3 and 1 (i.e., the split partitions 5 and 1) in the dynamic index have already been updated in step S32. The data belonging to partition 4 remains unchanged, so the attributes of partition 4 remain unchanged, thus completing the update of the dynamic index. In this case, the dynamic index includes four partitions (partitions 1, 2, 3, and 4) and their corresponding attributes (core and density). The data structure of the dynamic index can be found in Table 1 and related descriptions above.
[0122] In the example described above, with an existing dynamic index, for each received data item, it is possible to choose to assign the data to an existing partition, followed by a judgment on whether the partition should be split and whether the split partitions should be merged. That is, if step S31 splits the partition with the strongest correlation to the data into two partitions, the total number of partitions after receiving a data item and performing partition splitting may be exactly equal to the partition number threshold. In this case, if the next data item is received and the data processing method of this application determines that the partition can be split, the total number of partitions after splitting may exceed the partition number threshold, and the total number of partitions will be one more than the partition number threshold. Therefore, after executing step S34 to merge the two partitions with the strongest correlation into one partition, the total number of partitions can be equal to the partition number threshold, without needing to compare the total number of merged partitions with the partition number threshold.
[0123] In one possible implementation, after splitting the partition with the strongest data correlation in the dynamic index into multiple partitions in step S31, the core and density of each partition under the current partitioning method can be calculated first, without initially calculating the core of each partition. The total number of partitions after splitting is then calculated, and the relationship between the total number of partitions and a preset partition number threshold is used to determine whether the partitions in the dynamic index need to be merged. If merging is not required, for example, if the total number of partitions is less than or equal to the partition number threshold, the core and density of the split partitions are then calculated, thus completing the update of the dynamic index. If merging is required, for example, if the total number of partitions is greater than the partition number threshold, the core of the split partitions can be calculated first, and steps S33-S34 can be executed to determine the proximity of each pair of partitions after splitting, merge the partitions, and then calculate the core and density of the merged partitions, thus completing the update of the dynamic index.
[0124] Those skilled in the art should understand that, on the one hand, the core and density of a partition can be calculated in real time as the partitioning method changes; on the other hand, after receiving new data and determining the partition with the strongest correlation to the data, there may be changes in the partitioning method such as partition splitting and partition merging. If it is not determined whether the changed partitioning method is a suitable partitioning method (for example, the partition density is less than or equal to a first threshold and / or the total number of partitions is less than or equal to a threshold for the number of partitions), the core or density in the changed partitioning parameters may not be calculated immediately. As long as it can satisfy the requirement that a suitable partitioning method can be determined based on the core or density in the changed partitioning parameters, and the core and density in the partitioning parameters can be calculated after the suitable partitioning method is determined, this application does not restrict whether the core and density of a partition are calculated in real time as the partitioning method changes, or whether the core and density of a partition are calculated simultaneously.
[0125] In one possible implementation, if step S31 splits the selected data partition into multiple partitions, after step S35, the total number of merged partitions can be determined. If the total number of partitions still exceeds the partition number threshold, the above-mentioned proximity calculation and partition merging operation can be repeated until it is determined that the number of merged partitions is less than or equal to the partition number threshold. The partition merging method can be set according to the partition splitting method; this application does not limit the specific partition merging method.
[0126] Those skilled in the art should understand that, in addition to setting the number of partitions based on the computing power of the cluster nodes, the number of partitions threshold can also be set based on other parameters, such as the computing power of the receiving device. As long as it can be set in advance before the receiving device receives the streaming data, this application does not restrict the setting method of the number of partitions threshold.
[0127] In this way, when a dynamic index exists on the receiving device, data splitting and dynamic index updates can be completed by combining the dynamic index. This allows the dynamic index to participate in subsequent data splitting.
[0128] The following is combined Figure 12 This document presents an example of a data processing method according to embodiments of this application when a dynamic index does not exist on the receiving device.
[0129] Figure 12 An example of a data processing method according to an embodiment of this application is shown. For example... Figure 12 As shown, in one possible implementation, the data processing method of this application embodiment is executed by a receiving device, which can be a terminal device or a server. The method may include:
[0130] S11, Receive data from the transmitting device. The specific implementation method can be found in step S21 above.
[0131] S12, determine whether a dynamic index already exists on the receiving device.
[0132] S13, create a dynamic index if no dynamic index exists.
[0133] For example, the data received in step S11 could be the first data received after the start of receiving streaming data. At this point, the receiving device might not have a dynamic index. In this case, a dynamic index can be created, which will contain only one partition. The mapping between partition 1 and cluster nodes can be set in the same way as the mapping between partition 5 and cluster nodes described above. Partition 1 can be pre-assigned to a specific cluster node, or it can be assigned to the cluster node with the fewest corresponding partitions. If multiple cluster nodes do not have corresponding partitions, any one of them can be chosen to correspond to partition 1.
[0134] S14. Based on the correspondence between partitions and cluster nodes, the data is output to the cluster node corresponding to the partition.
[0135] For example, partition 1 may correspond to the first cluster node. Since the dynamic index only includes one partition, the receiving device can output the data received in step S11 to the first cluster node corresponding to partition 1.
[0136] S15, Update the partition attributes in the dynamic index according to the data.
[0137] For example, since the dynamic index includes only one partition, this single partition can be chosen as the partition to which the data belongs, such as partition 1. In this case, there is one and only one piece of data (the first piece of data) belonging to partition 1, and the attributes of partition 1 can be determined based on this one piece of data (the first piece of data). Taking each piece of streaming data as having two dimensions a and b, and the first piece of data being (a1, b1) as an example, the position of the first piece of data in a two-dimensional coordinate system with dimensions a and b as axes is also (a1, b1). Since there is one and only one piece of data in partition 1, the core M1 of partition 1 can be, for example, the position of the first piece of data (a1, b1), and the density D1 of partition 1 can be, for example, 0. After determining the core M1 and density D1, the core M1 and density D1 can be stored correspondingly with partition 1, completing the update of the dynamic index. In this case, the dynamic index includes one partition (partition 1) and the corresponding attributes of this partition (core M1, density D1). The data structure for dynamic indexes can be found in Table 1 above and related descriptions.
[0138] In this way, data splitting and the creation and updating of dynamic indexes can be completed even when dynamic indexes are not present on the receiving device. This allows dynamic indexes to participate in subsequent data splitting.
[0139] In one possible implementation, after steps S24 and S14, data can be locally clustered on each cluster node. This process can be implemented based on existing technologies such as the K-MEANS clustering algorithm. The clustering algorithm used by each cluster node can be pre-set, and based on the pre-set clustering algorithm, the receiving device can perform clustering calculations to determine partition attributes, ensuring that the receiving device's clustering calculations use the same algorithm as the cluster nodes' local clustering.
[0140] Local clustering typically clusters a batch of data, which can be data received by cluster nodes from receiving devices within a time window. In other words, when clustering a piece of data flowing into a cluster node, the node can combine historical inflow data before the inflow time of that data with future inflow data after the inflow time to complete the local clustering. The data targeted by local clustering can, for example, include all data flowing into the cluster node from the beginning to the end of a time window. Local clustering can, for example, be performed at the end of the time window. Once the clustering results are obtained, they can be reported to the management node (not shown) or other devices in the distributed system, such as receiving devices.
[0141] Figure 13 An example of a time window according to an embodiment of this application is shown. Common selectable window formats include scrolling windows (…). Figure 13 The right side of the "scrolling window" (windows 1-4) and the sliding window ( Figure 13 (Window 1-Window 3 on the right side of the "sliding window"). When using a rolling window, the data (event stream) flowing into the cluster node within each time window has a different inflow time than the data flowing into the same cluster node in other time windows. When using a sliding window, some data flowing into the cluster node within each time window shares the same inflow time as the data flowing into the same cluster node in the preceding and following time windows. With a rolling window, the data in a given time window can be cleared after clustering, reducing data storage costs for the cluster node. With a sliding window, any data point is clustered together with the two data points before and after it, reducing the probability of data from the same community being clustered separately in two different time windows, leading to data from the same community failing to belong to the same partition and flow to the same cluster node, thus improving the accuracy of the clustering results.
[0142] In one possible implementation, streaming data continuously enters the receiving device, causing the device's dynamic index to be constantly updated. Based on this updated index, newly arriving data is then routed. This process ensures that data arriving earlier continuously influences the routing of data arriving later. If the earlier data is more timely than the later data (e.g., the time difference is small), the accuracy of the dynamic index used to guide the routing of later data is considered high. Conversely, if the earlier data is less timely than the later data (e.g., the time difference is large), the accuracy of the dynamic index used to guide the routing of later data is considered low. In this case, the dynamic index can be modified to improve its accuracy in guiding data routing.
[0143] For example, if some or all of the data used in the creation or updating of a dynamic index is too old for the current data received by the receiving device, then this outdated data should not affect the current data flow. This can be achieved by processing the dynamic index to obtain a processed version that removes the influence of outdated data.
[0144] In addition to the partitions and attributes described in Table 1 above, the dynamic index can also indicate, for example, the reception time of data belonging to each partition. The processing of the dynamic index can be triggered by user commands or initiated by the receiving device. When triggered by the user, the user's command can, for example, instruct the receiving device to rebuild and update the index. According to the user's command, the receiving device can determine a time point that is earlier than the time of the most recently received data, based on the time of the most recently received data and a preset time threshold, as the time start point. The time of the most recently received data is then used as the time end point. The dynamic index is recreated based on the first data received after this time start point, and the dynamic index is updated based on the data received by the receiving device between the time start point and the time end point. Each data entry is updated once. For specific creation and update methods, please refer to [link to documentation]. Figure 12 and Figure 5 Alternatively, the data received by the receiving device within the time frame from the start to the end of the time period can be clustered, and the clustering results can be used as partitions. The parameters of the partitions can then be determined using the exemplary method described above. In this case, the reconstructed and updated dynamic index is a dynamic index that eliminates the influence of old data. When using this index for data splitting, the data splitting will not be affected by excessively old data.
[0145] When the processing of dynamic indexes is initiated by the receiving device, the receiving device can trigger the processing of dynamic indexes at a certain frequency. The specific processing method can be the same as when it is triggered by user commands.
[0146] Those skilled in the art will understand that there are multiple ways to eliminate the impact of old data in dynamic indexes, and this application does not limit such options.
[0147] In one possible implementation, when the type of streaming data to be received is known, and the receiving device has previously received the same type of streaming data and performed data splitting, or can obtain the same type of non-streaming data, a dynamic index can be created before receiving the streaming data based on previously received streaming or non-streaming data of the same type. For example, if the streaming data to be received is information about students enrolled in 2021, and the receiving device has previously received and created a dynamic index for information about students enrolled in 2020, then the dynamic index created for the 2020 students can be used as the initial dynamic index when receiving information about students enrolled in 2021. Based on the newly received data, the data splitting is performed, and the dynamic index is updated. Alternatively, the initial dynamic index can be created using the clustering results for information about students enrolled in 2020. Whether an old dynamic index or old data can be used in the splitting process of new data can be determined based on the timeliness of the data. Provided that the old dynamic index or old data still has timeliness, it can be used in the splitting process of new data. This method can improve the accuracy of receiving devices in using dynamic indexes to distribute data.
[0148] In one possible implementation, the timeliness of old dynamic indexes or old data can be determined proactively by the receiving device or triggered by a user command. The specific implementation for determining timeliness is described above and will not be repeated here.
[0149] According to the data processing method of this application embodiment, by determining the partition to which the data belongs, the distributed node to which the data flows is determined. This ensures that data clustered in the same community on the distributed node flows to the same node, solving the clustering segmentation problem in distributed systems, increasing the probability that data from the same community flows into the same node, and optimizing the clustering effect. The computational cost of determining the partition to which the data belongs in this application embodiment is far less than the computational cost of clustering the data, reducing data processing costs. At the same time, the solution is simple, requiring no additional feature or rule design, reducing the complexity of data distribution. The data partition determination mechanism (dynamic index) of this application embodiment is updated in real time with the data, thus exhibiting strong adaptability to different application scenarios. Furthermore, by dynamically updating the dynamic index, the accuracy of partitioning can be improved, avoiding the problem of uncorrectable errors inherent in fixed partitioning mechanisms.
[0150] Figure 14 An exemplary flowchart of a data processing method according to an embodiment of this application is shown. The method is applied to a distributed system, which includes a flow guidance node and multiple processing nodes. The flow guidance node stores a dynamic index, which includes multiple partitions, each partition corresponding to a processing node. The method includes:
[0151] S1401, the routing guidance node receives data and selects the partition to which the data belongs from the plurality of partitions according to the parameters of each partition in the dynamic index;
[0152] S1402, the diversion guidance node outputs the data to the processing node corresponding to the selected partition;
[0153] S1403, the traffic distribution guidance node updates the dynamic index based on the data belonging to the selected partition.
[0154] The data processing method in this application embodiment can be referred to above. Figure 5 As described above, the data processing method's routing guidance node can correspond to the receiving device mentioned in the related description, and the processing node can correspond to the cluster node mentioned in the related description. Optionally, the data processing method's routing guidance node can also be one of the cluster nodes.
[0155] An example of step S1401 can be found above and Figure 5 Examples of steps S21, S22, S23, and step S1402 can be found above and in the following text. Figure 5 Step S24, where the dynamic index can be found in [reference needed]. Figure 6 For example, the selected partition could be, for instance, Figure 7Partition 1 in the dynamic index can have parameters that include, for example, the partition attributes described in step S23 above, i.e. Figure 6 The core and density of the material. An example of step S1403 can be found above and... Figure 5 Step S25 in the process.
[0156] According to the data processing method of this application embodiment, by using the partition parameters in the dynamic index, the partition to which the data belongs can be selected from multiple partitions in the dynamic index. This ensures that when data is output to the processing node corresponding to the selected partition according to the correspondence between the partition and the processing node, there is already data belonging to the selected partition on that processing node, thereby enabling data belonging to the same partition to be distributed to the same node. After selecting the partition to which the data belongs, the dynamic index can be updated according to the data belonging to the selected partition. This further improves the accuracy of data distribution when subsequent received data is output to the processing node according to the updated dynamic index, avoiding the problem of uncorrectable errors in fixed partitioning mechanisms, increasing the probability of data belonging to the same community flowing into the same node, reducing the adverse effects of the "clustering fragmentation problem", and improving the streaming clustering effect and adaptability to different application scenarios.
[0157] In one possible implementation, each partition's parameters include a core and a density, where the core indicates the partition's central location, and the density indicates how closely the data belonging to the partition is packed together. Examples of core and density can be found above. Figure 4 The core M and density D are mentioned in the relevant description.
[0158] In this way, partition parameters can be used to reflect relevant information about the data in the partition, ensuring the accuracy of selecting the partition to which the data belongs based on the partition parameters.
[0159] In one possible implementation, selecting the partition to which the data belongs from the plurality of partitions based on the parameters of each partition in the dynamic index includes: determining the proximity of the data to the core of each of the plurality of partitions; and selecting the partition whose core is closest to the data as the partition to which the data belongs.
[0160] An example of determining the degree of proximity can be found above. Figure 7 And related descriptions.
[0161] In this way, the partition affiliation of received data can be determined. The closer the data is to the core of the partition, the more accurate the data's affiliation to that partition becomes. Furthermore, determining the proximity between the partition core and the data is relatively simple, reducing the complexity of selecting the partition to which the data belongs.
[0162] In one possible implementation, updating the dynamic index based on data belonging to the selected partition includes: updating the core and density of the selected partition based on data belonging to the selected partition.
[0163] Its exemplary implementation can be referred to the above and Figure 8 Related descriptions.
[0164] In this way, partition parameters can be updated without changing the partitioning method, making dynamic index updates simpler and more convenient. After selecting a partition to which data belongs, the data belonging to that partition changes. The core and density of the selected partition are updated based on the changed data. This allows the selection of a new partition to belong to when new data is received subsequently, based on the updated core and density. In other words, the updated core and density are used to determine whether new data is highly similar to the selected partition, improving the accuracy of data partitioning.
[0165] In one possible implementation, updating the dynamic index based on data belonging to the selected partition includes: updating the density of the selected partition based on data belonging to the selected partition; determining that the updated density is greater than a first threshold, splitting the selected partition into multiple partitions in the dynamic index; and updating the core and density of each partition based on data belonging to each partition in the dynamic index.
[0166] An example of the updated density can be found in the density D' section above, and an example of the first threshold can be found in the first threshold U section above. If the updated density is determined to be greater than the first threshold, the selected partition is split into multiple partitions in the dynamic index, as described above. Figure 9 In step S31, the selected partition can be, for example, Figure 9 Partition 1 in the middle can be split into multiple partitions, for example, it could be... Figure 9 For an example of updating the core and density of partitions 1-1 and 1-2 based on the data belonging to each partition in the dynamic index, see the above and [link to example]. Figure 9 Step S32 in the process.
[0167] By changing the partitioning method through partition splitting, the partition parameters are also changed, thus enabling the dynamic index to be updated. Since partition splitting is performed when the density does not meet the threshold condition, the resulting dynamic index with better partition density after splitting is updated, making the selection of the partition to which data belongs based on the updated dynamic index more accurate.
[0168] In one possible implementation, updating the dynamic index based on the data belonging to the selected partitions further includes: determining that the total number of partitions in the dynamic index is greater than a second threshold, selecting multiple partitions in the dynamic index for merging; and updating the core and density of the merged partitions based on the data belonging to the merged partitions.
[0169] An example of the second threshold can be found in the partition number threshold W above. An example of merging multiple partitions in the dynamic index when the total number of partitions in the dynamic index is greater than the second threshold can be found in the above and... Figure 11 For an example of step S34 in the text, updating the core and density of the merged partition based on the data belonging to the merged partition, please refer to the above and... Figure 11 Step S35 in the process.
[0170] In this way, both the partitioning method and partitioning parameters change, enabling the dynamic index to be updated. Since partition merging is performed when the total number of partitions does not meet a threshold, the resulting dynamic index, updated through merging, has a more optimal number of partitions. The total number of partitions is related to the number of partitioning parameters in the dynamic index; a more optimal number of partitions results in lower time and computational costs for selecting the partition to which the data belongs based on the parameters in the dynamic index. This allows for high computational efficiency while maintaining sufficient accuracy when selecting the partition to which data belongs based on the updated dynamic index.
[0171] In one possible implementation, merging multiple partitions in the dynamic index includes: determining the proximity of every two partitions in the dynamic index; and merging the two closest partitions into one partition.
[0172] For an example of determining the proximity of every two partitions in the dynamic index, please refer to the above and Figure 10 Step S33. Select the partition merging method that merges the two closest partitions into one partition. This can be done when the selected partition is split into two partitions.
[0173] In this way, partition merging can be completed, reducing the total number of partitions after merging, so as to get closer to the threshold requirement.
[0174] In one possible implementation, the method further includes:
[0175] Determine whether the data belonging to the partition of the dynamic index is time-sensitive; when some or all of the data loses its timeliness, delete the dynamic index, or update the dynamic index based on the time-sensitive data.
[0176] For examples of data timeliness and updating dynamic indexes based on time-sensitive data, please refer to the above text. Figure 13 The following is a related description.
[0177] In this way, the negative impact of old data on dynamic indexes can be eliminated, further improving the accuracy of selecting the partition to which data belongs based on the dynamic index.
[0178] Figure 15 This diagram illustrates an exemplary structure of a data processing apparatus according to an embodiment of the present application.
[0179] Embodiments of this application provide a data processing apparatus applied in a distributed system. The distributed system includes a flow guidance node and multiple processing nodes. The flow guidance node stores a dynamic index, which includes multiple partitions, each partition corresponding to a processing node. Figure 15 As shown, the device includes:
[0180] The partition determination module 101 is set on the traffic distribution guidance node and is used to receive data and select the partition to which the data belongs from the plurality of partitions according to the parameters of each partition in the dynamic index.
[0181] Data output module 102 is disposed on the diversion guidance node and is used to output the data to the processing node corresponding to the selected partition;
[0182] The index update module 103 is set on the routing guidance node and is used to update the dynamic index based on the data belonging to the selected partition.
[0183] In one possible implementation, the parameters for each partition include a core and a density, wherein the core indicates the central location of the partition and the density indicates the degree of closeness between the data belonging to the partition.
[0184] In one possible implementation, selecting the partition to which the data belongs from the plurality of partitions based on the parameters of each partition in the dynamic index includes: determining the proximity of the data to the core of each of the plurality of partitions; and selecting the partition whose core is closest to the data as the partition to which the data belongs.
[0185] In one possible implementation, updating the dynamic index based on data belonging to the selected partition includes: updating the core and density of the selected partition based on data belonging to the selected partition.
[0186] In one possible implementation, updating the dynamic index based on data belonging to the selected partition includes: updating the density of the selected partition based on data belonging to the selected partition; determining that the updated density is greater than a first threshold, splitting the selected partition into multiple partitions in the dynamic index; and updating the core and density of each partition based on data belonging to each partition in the dynamic index.
[0187] In one possible implementation, updating the dynamic index based on the data belonging to the selected partitions further includes: determining that the total number of partitions in the dynamic index is greater than a second threshold, selecting multiple partitions in the dynamic index for merging; and updating the core and density of the merged partitions based on the data belonging to the merged partitions.
[0188] In one possible implementation, merging multiple partitions in the dynamic index includes: determining the proximity of every two partitions in the dynamic index; and merging the two closest partitions into one partition.
[0189] In one possible implementation, the device further includes:
[0190] The timeliness determination module, set on the diversion guidance node, is used to determine whether the data belonging to the partition of the dynamic index has timeliness.
[0191] An index processing module, located on the diversion guidance node, is used to delete the dynamic index when some or all of the data loses its timeliness, or to update the dynamic index based on the data that has timeliness.
[0192] Embodiments of this application provide a data processing apparatus, including: a processor and a memory for storing processor-executable instructions; wherein the processor is configured to implement the above-described method when executing the instructions.
[0193] Embodiments of this application provide a non-volatile computer-readable storage medium storing computer program instructions thereon, which, when executed by a processor, implement the above-described method.
[0194] Embodiments of this application provide a computer program product including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code, wherein when the computer-readable code is run in a processor of an electronic device, the processor in the electronic device performs the above-described method.
[0195] Figure 16 This diagram illustrates an exemplary structure of a data processing apparatus according to an embodiment of the present application.
[0196] Data processing devices may include at least one of the following: mobile phones, foldable electronic devices, tablet computers, desktop computers, laptop computers, handheld computers, laptops, ultra-mobile personal computers (UMPCs), netbooks, cellular phones, personal digital assistants (PDAs), augmented reality (AR) devices, virtual reality (VR) devices, artificial intelligence (AI) devices, wearable devices, in-vehicle devices, smart home devices, or smart city devices, and server devices. This application does not impose any special limitations on the specific type of data processing device.
[0197] The data processing device may include a processor 110, a memory 121, and a communication module 160. It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the data processing device. In other embodiments of this application, the data processing device may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.
[0198] Processor 110 may include one or more processing units, such as application processors (APs), modem processors, graphics processing units (GPUs), image signal processors (ISPs), controllers, video codecs, digital signal processors (DSPs), baseband processors, and / or neural network processing units (NPUs). These different processing units may be independent devices or integrated into one or more processors.
[0199] The processor can generate operation control signals based on the instruction opcode and timing signals to control the instruction fetching and execution.
[0200] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 may be a cache memory. This memory can store instructions or data that the processor 110 has used or that are used frequently. If the processor 110 needs to use the instruction or data, it can directly retrieve it from this memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.
[0201] Memory 121 can be used to store computer executable program code, including instructions. Memory 121 may include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as data acquisition functions, data transfer functions, etc.). The data storage area may store data created during the use of the data processing device (such as dynamic indexes, etc.). In addition, memory 121 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc. Processor 110 executes various functional methods of the data processing device or the above-described data processing methods by running instructions stored in memory 121 and / or instructions stored in memory disposed in the processor.
[0202] The communication module 160 can be used to receive streaming data from other devices or equipment (such as the transmitting device in this embodiment) via wireless / wired communication, and to output streaming data to other devices or equipment (such as the cluster node in this embodiment). For example, it can provide wireless communication solutions including WLAN (such as Wi-Fi networks), Bluetooth (BT), Global Navigation Satellite System (GNSS), Frequency Modulation (FM), Near Field Communication (NFC), and Infrared (IR).
[0203] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example—but not limited to—electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), electrically programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), compact disc read-only memory (CD-ROM), digital video disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination of the foregoing.
[0204] The computer-readable program instructions or code described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.
[0205] The computer program instructions used to perform the operations of this application may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages such as Smalltalk, C++, etc., and conventional procedural programming languages such as "C" or similar languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuits, such as programmable logic circuits, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), are personalized by utilizing state information from computer-readable program instructions. These electronic circuits can execute computer-readable program instructions to implement various aspects of this application.
[0206] Various aspects of this application are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.
[0207] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.
[0208] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.
[0209] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved.
[0210] It should also be noted that each block in the block diagram and / or flowchart, as well as combinations of blocks in the block diagram and / or flowchart, can be implemented using hardware (such as circuits or ASICs (Application Specific Integrated Circuits)) that performs the corresponding function or action, or using a combination of hardware and software, such as firmware.
[0211] Although the invention has been described herein in conjunction with various embodiments, those skilled in the art will understand and implement other variations of the disclosed embodiments by reviewing the accompanying drawings, disclosure, and appended claims in carrying out the claimed invention. In the claims, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude a plurality. A single processor or other unit can implement several functions listed in the claims. While different dependent claims may recite certain measures, this does not mean that these measures cannot be combined to produce good results.
[0212] The various embodiments of this application have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or improvement of the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.
Claims
1. A data processing method, said method being applied to a distributed system, said distributed system comprising a flow guidance node and multiple processing nodes, wherein, The traffic routing guidance node stores a dynamic index, which includes multiple partitions, each partition corresponding to a processing node. The method comprises: The routing guidance node receives data and selects the partition to which the data belongs from the plurality of partitions based on the parameters of each partition in the dynamic index. The data routing guidance node outputs the data to the processing node corresponding to the selected partition; The traffic distribution guidance node updates the dynamic index based on the data belonging to the selected partition; The step of updating the dynamic index based on the data belonging to the selected partition includes: updating the density of the selected partition based on the data belonging to the selected partition; determining that the updated density is greater than a first threshold, splitting the selected partition into multiple partitions in the dynamic index; and updating the core and density of each partition based on the data belonging to each partition in the dynamic index.
2. The method according to claim 1, characterized in that, Each partition's parameters include core and density. The core indicates the center location of the partition, and the density indicates how closely the data belonging to the partition is packed together.
3. The method according to claim 2, characterized in that, The step of selecting the partition to which the data belongs from the plurality of partitions based on the parameters of each partition in the dynamic index includes: Determine the proximity of the data to the core of each of the plurality of partitions; Select the partition whose core data is closest to the data as the partition to which the data belongs.
4. The method according to any one of claims 1-3, characterized in that, Updating the dynamic index based on data belonging to the selected partition includes: Update the core and density of the selected partitions based on the data belonging to the selected partitions.
5. The method according to claim 1, characterized in that, The step of updating the dynamic index based on data belonging to the selected partition further includes: If the total number of partitions in the dynamic index is greater than the second threshold, multiple partitions in the dynamic index are selected for merging. Update the core and density of the merged partition based on the data belonging to the merged partition.
6. The method according to claim 5, characterized in that, Multiple partitions in the dynamic index are selected for merging, including: Determine the proximity of every two partitions in the dynamic index; Select the two closest partitions and merge them into one partition.
7. The method according to any one of claims 1-3, characterized in that, The method further includes: Determine whether the data belonging to the partition of the dynamic index is time-sensitive; When some or all of the data becomes outdated, the dynamic index is deleted, or the dynamic index is updated based on the data that is outdated.
8. A data processing apparatus, the apparatus being applied to a distributed system, the distributed system comprising a flow guidance node and multiple processing nodes, wherein, The traffic routing guidance node stores a dynamic index, which includes multiple partitions, each partition corresponding to a processing node. The device comprises: A partition determination module is set on the traffic distribution guidance node to receive data and select the partition to which the data belongs from the multiple partitions according to the parameters of each partition in the dynamic index. A data output module is installed on the diversion guidance node and is used to output the data to the processing node corresponding to the selected partition; An index update module, configured on the traffic splitting guidance node, is used to update the dynamic index based on data belonging to the selected partitions. Updating the dynamic index based on data belonging to the selected partitions includes: updating the density of the selected partitions based on the data belonging to the selected partitions; determining that the updated density is greater than a first threshold, splitting the selected partitions into multiple partitions in the dynamic index; and updating the core and density of each partition based on data belonging to each partition in the dynamic index.
9. The apparatus according to claim 8, characterized in that, Each partition's parameters include core and density. The core indicates the center location of the partition, and the density indicates how closely the data belonging to the partition is packed together.
10. The apparatus according to claim 9, characterized in that, The step of selecting the partition to which the data belongs from the plurality of partitions based on the parameters of each partition in the dynamic index includes: Determine the proximity of the data to the core of each of the plurality of partitions; Select the partition whose core data is closest to the data as the partition to which the data belongs.
11. The apparatus according to any one of claims 8-10, characterized in that, Updating the dynamic index based on data belonging to the selected partition includes: Update the core and density of the selected partitions based on the data belonging to the selected partitions.
12. The apparatus according to claim 8, characterized in that, The step of updating the dynamic index based on data belonging to the selected partition further includes: If the total number of partitions in the dynamic index is greater than the second threshold, multiple partitions in the dynamic index are selected for merging. Update the core and density of the merged partition based on the data belonging to the merged partition.
13. The apparatus according to claim 12, characterized in that, Multiple partitions in the dynamic index are selected for merging, including: Determine the proximity of every two partitions in the dynamic index; Select the two closest partitions and merge them into one partition.
14. The apparatus according to any one of claims 8-10, characterized in that, The device further includes: The timeliness determination module, set on the diversion guidance node, is used to determine whether the data belonging to the partition of the dynamic index has timeliness. An index processing module, located on the diversion guidance node, is used to delete the dynamic index when some or all of the data loses its timeliness, or to update the dynamic index based on the data that has timeliness.
15. A data processing apparatus, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to implement the method of any one of claims 1-7 when executing the instructions.
16. A non-volatile computer-readable storage medium storing computer program instructions thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1-7.
17. A computer program product, characterized in that, Includes computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code, wherein when the computer-readable code is executed in an electronic device, the processor in the electronic device performs the method according to any one of claims 1-7.
Citation Information
Patent Citations
Data processing method and management node
CN106682215A
Construction method and device of news content full-text retrieval engine
CN108804594A
