Data processing method and computer equipment

Abstract

The invention provides a data processing method and computer equipment, and the method comprises the steps: determining a data representation tree corresponding to any target data in a data stream, carrying out the node recursion processing from a root node of the data representation tree, and determining the distance between the target data and a corresponding node, and if the data represents that the variance of the node without the child node in the tree is greater than a corresponding variance threshold value, taking the node as a father node, and splitting the node into a corresponding child node for storing the data element, until the node without the child node and with the minimum distance is determined as a target node for storing the data element of the target data, updating the variance of the target node, and if the data represents that the variance of the node without the child node in the tree is greater than the corresponding variance threshold value. The node with the minimum distance in each layer can be determined based on node recursion, so that data with similar characteristics can be recursively transmitted to child nodes in the same branch, the relationship between the data can be better reflected, and the variance is taken as the basis of node splitting, so that each node is the data meeting the data distribution requirement, and the data distribution efficiency is improved. And the data distribution characteristics can be better reflected.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a data processing method and computer equipment. Background Technology

[0002] In fields such as big data analytics, pattern recognition, and machine learning, it is often necessary to process and represent the continuously arriving data streams in order to quickly extract data features and perform data classification or clustering. Therefore, how to process and represent data in data streams to better reflect data features and relationships between data is a current research direction in the field. Summary of the Invention

[0003] This application provides a data processing method and computer device. For data that needs to be represented, the data representation tree corresponding to the data is recursively processed starting from the root node. The root node is used as the first recursive detection node. The child node with the smallest distance to the recursive detection node is dynamically selected until the node with the smallest distance and no child nodes is determined. This node is then used as the target node for storing the data element. The variance of the target node is updated, which is also the variance of the node with no child nodes. Based on the variance of the node with no child nodes, it is determined whether it is necessary to split the node into a new child node. This method of splitting child nodes based on variance ensures that data with small cumulative variance can always be stored in one node, which better reflects the distribution characteristics of the data. Only when the variance is greater than the variance threshold and the data distribution in the node is relatively scattered, indicating that the data correlation is getting weaker, is it necessary to split the child node so that the next data in the data stream can be stored in the new target node based on the recursive processing process. This dynamic node splitting method, and the method of node splitting based on variance, can update the data distribution structure in real time based on the data distribution characteristics, obtain new branch numbers, and make the shape of the data representation tree closer to the data distribution characteristics, and better reflect the relationship between data.

[0004] To address the aforementioned technical problems, in a first aspect, embodiments of this application provide a data processing method. This method includes: determining a data representation tree corresponding to target data, where the target data is any data in a data stream requiring data representation processing; starting from the root node of the data representation tree, performing recursive node processing, using the root node as the first recursive detection node, determining whether the recursive detection node has child nodes; if the recursive detection node has child nodes, calculating the distance between the target data and each child node corresponding to the recursive detection node, selecting the child node with the smallest distance as the next recursive detection node; if the recursive detection node has no child nodes, using the recursive detection node as the target node corresponding to the target data, storing the data elements of the target data in the target node, and updating the node information of the target node, the node information including variance; and, if the variance of a node without child nodes in the data representation tree is greater than a corresponding variance threshold, using the node without child nodes as the parent node, splitting off corresponding child nodes for storing data elements of the data in the data stream.

[0005] Using the above technical solution, for any data in the data stream, starting from the root node of the data representation tree, node recursive processing is performed to determine the child node with the smallest distance for each recursive detection node, until the node with the smallest distance and no child nodes is obtained, which is used as the target node for storing the data element of the target data, and the variance of the target node is updated. If the variance of the target node is greater than the corresponding variance threshold, the target node is used as the parent node and the corresponding child nodes are split off. Thus, based on distance, the node with the smallest distance in each level of the data representation tree is determined until the target node with the smallest distance is found. The smaller the distance, the more similar the data is to the features of the corresponding node. This allows data with similar features to be recursively sent to the child nodes corresponding to the same recursive detection node, which better reflects the relationship between the data. Furthermore, the smaller the variance, the more concentrated the data distribution; the larger the variance, the more dispersed the data distribution. Using variance as the basis for node splitting, when the data distribution is relatively dispersed, node splitting is triggered for the storage of subsequent data in the data stream. This ensures that the data elements in each node meet the data distribution requirements, better reflecting the data distribution characteristics. Moreover, the data elements stored in the same node are data elements with similar features, which better reflects the relationship between the data.

[0006] In one possible implementation of the first aspect above, calculating the distance between the target data and each child node corresponding to the recursive detection node includes: determining the current mean of each child node corresponding to the recursive detection node; and calculating the distance between the target data and each child node corresponding to the recursive detection node based on the target data and the current mean of each child node corresponding to the recursive detection node.

[0007] Using the above technical solution, the node mean can reflect the characteristics of the data covered by the node. The closer the data is to the mean, the higher the degree of fit with the characteristics of other data in the node. The distance between the data and the node is determined based on the node mean. The process is recursive, and the node with the smallest distance is selected at each level until the node with the smallest distance and no child nodes is selected. This allows the data to be stored in nodes with similar characteristics. The characteristics of the data corresponding to each node on a node branch are interrelated, which can better reflect the characteristic relationship between the data and better manage data with similar characteristics.

[0008] In one possible implementation of the first aspect above, the node information also includes a data count and a mean. Updating the node information of the target node includes: incrementing the data count of the target node by one to update the data count of the target node; determining the current mean of the target node, and determining the target mean of the target node based on the target data, the current mean, and the updated data count; and updating the mean of the target node to the target mean.

[0009] In one possible implementation of the first aspect above, the target mean of the target node is determined based on the target data, the current mean, and the updated data count, including obtaining the target mean of the target node in the following manner:

[0010]

[0011] in, The target mean of the target node. This represents the current mean of the target node. For target data, Count the number of data items updated for the target node.

[0012] By adopting the above technical solution, the data count of the node storing the data is updated every time data is stored, which facilitates the statistical analysis of the data count of each node. Based on the target data, the current average, and the updated data count, the target average of the target node is determined. The average of the target node is updated to the target average based on the stored data, so that the average of the node can be updated in real time with data storage, which is more conducive to reflecting the characteristic relationship of the data stored in the node.

[0013] In one possible implementation of the first aspect above, updating the node information of the target node further includes: determining the current variance of the target node; determining the target variance of the target node based on the current variance of the target node, the target data, the current mean of the target node, the target mean of the target node, and the updated data count of the target node; and updating the variance of the target node to the target variance.

[0014] In one possible implementation of the first aspect above, the target variance of the target node is determined based on the current variance of the target node, the target data, the current mean of the target node, the target mean of the target node, and the updated data count of the target node. This includes obtaining the target variance of the target node in the following manner:

[0015]

[0016] in, Let V be the target variance of the target node. Let be the current variance of the target node. The target mean of the target node. This represents the current mean of the target node. For target data, Count the number of data items updated for the target node.

[0017] By adopting the above technical solution, a variance update is performed every time data is stored, so that the data distribution characteristics of the data stored in the node can be better seen based on the variance. Furthermore, the variance is used as the basis for node splitting. As long as the variance threshold is met, child nodes can be automatically split. In this way, the data representation tree can be dynamically expanded according to the data distribution characteristics, which facilitates real-time data storage and better reflects the data distribution of each node.

[0018] In one possible implementation of the first aspect above, the current mean and current variance of nodes in the data representation tree that do not store data elements are empty, or the current mean and current variance of nodes in the data representation tree that do not store data elements are generated based on a clustering method or a random generation method.

[0019] In one possible implementation of the first aspect above, the number of child node splits is determined based on the attribute information of the data representation tree, which includes at least one of the following: dimension, data distribution, and data distribution requirements.

[0020] By adopting the above technical solution, the number of child node splits can be determined based on at least one of the following information: data dimension, data distribution, and data distribution requirements. This enables dynamic node splitting, making the node layer of the data representation tree more reflective of data dimensions and the number of nodes more in line with data distribution and requirements.

[0021] In one possible implementation of the first aspect described above, the data representation tree is a multi-branch tree structure.

[0022] By adopting the above technical solution and storing data based on a multi-branch tree structure, the data distribution characteristics can be better reflected.

[0023] Secondly, this application also discloses a computer device, including: a processor and a memory communicatively connected to the processor; the memory stores computer execution instructions; the processor executes the computer execution instructions stored in the memory to enable the computer device to implement the data processing method provided by any of the implementations of the first aspect.

[0024] Thirdly, this application also discloses a computer-readable storage medium storing a computer program that can be executed by a computer device to implement the data processing method provided by any of the implementations of the first aspect.

[0025] Fourthly, this application also discloses a computer program product, including a computer program, which, when executed by a computer device, implements the data processing method provided by any of the implementations of the first aspect. Attached Figure Description

[0026] To more clearly illustrate the technical solution of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below.

[0027] Figure 1 A schematic flowchart of a data processing method provided in an embodiment of this application;

[0028] Figure 2 A schematic diagram of a data representation tree provided in an embodiment of this application;

[0029] Figure 3 A flowchart illustrating the process of determining the distance between target data and each child node, provided in an embodiment of this application;

[0030] Figure 4 A schematic diagram illustrating the process of updating node information for an embodiment of this application;

[0031] Figure 5 A schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0032] In fields such as big data analytics, pattern recognition, and machine learning, it is often necessary to represent and organize the continuously arriving data streams in order to quickly extract data features and classify or cluster the data. Traditional information representation methods include data processing based on grid structures with fixed partitions, binary tree partitioning methods with a preset number of levels, and clustering methods that perform batch processing. However, these methods have many shortcomings.

[0033] Fixed-partition mesh structures or tree structures with a preset number of layers struggle to dynamically adjust their depth and branch count based on data distribution. This leads to numerous idle grids or branches when the data volume is small, wasting resources. Conversely, when the data volume is large, once the fixed number of grids or branches is full, there is no way to represent the remaining data. Batch processing requires all data to be collected before processing can begin, which is insufficient for real-time streaming data. Therefore, traditional data processing methods often result in redundant data element storage, excessive computational load, and low computational efficiency when dealing with extremely large or high-dimensional data.

[0034] Based on this, this application proposes a method for dynamically and progressively constructing information representation structures under streaming data input conditions. This method should be able to gradually refine the representation granularity according to changes in data distribution in order to better reflect data characteristics and the relationships between data.

[0035] The data processing method proposed in this application, upon receiving the target data to be processed, determines the data representation tree corresponding to the target data. Starting from the root node of the data representation tree, it performs recursive node processing, using the root node as the first recursive detection node. It then determines whether the recursive detection node has child nodes. If the recursive detection node has child nodes, it calculates the distance between the target data and each child node corresponding to the recursive detection node, selecting the child node with the smallest distance as the next recursive detection node. If the recursive detection node has no child nodes, it is used as the target node corresponding to the target data. The data elements of the target data are stored in the target node, and the node information of the target node is updated. The node information includes variance. Furthermore, if the variance of a node without child nodes in the data representation tree is greater than the corresponding variance threshold, the node without child nodes is used as the parent node, and corresponding child nodes are split off to store the data elements of the data in the data stream.

[0036] Thus, based on distance, the node with the smallest distance in each level of the data representation tree is determined until the target node with the smallest distance is found. The smaller the distance, the more similar the data is to the features of the corresponding node. This allows data with similar features to be recursively sent to the child nodes corresponding to the same recursive detection node, which better reflects the relationship between the data. Furthermore, the smaller the variance, the more concentrated the data distribution; the larger the variance, the more dispersed the data distribution. Using variance as the basis for node splitting, when the data distribution is relatively dispersed, node splitting is triggered for the storage of subsequent data in the data stream. This ensures that the data elements in each node meet the data distribution requirements, better reflecting the data distribution characteristics. Moreover, the data elements stored in the same node are data elements with similar features, which better reflects the relationship between the data.

[0037] The data processing method provided in this application will be described in detail below.

[0038] like Figure 1 As shown, the data processing method provided in this application specifically includes the following steps.

[0039] S100, determine the data representation tree corresponding to the target data, where the target data is any data in the data stream that needs to be represented.

[0040] S200: Starting from the root node of the data representation tree, recursive node processing is performed. The root node is taken as the first recursive detection node. It is determined whether the recursive detection node has child nodes. If the recursive detection node has child nodes, the distance between the target data and each child node corresponding to the recursive detection node is calculated, and the child node with the smallest distance is selected as the next recursive detection node. If the recursive detection node has no child nodes, the recursive detection node is taken as the target node corresponding to the target data. The data elements of the target data are stored in the target node, and the node information of the target node is updated. The node information includes variance. Furthermore, if the variance of a node without child nodes in the data representation tree is greater than the corresponding variance threshold, the node without child nodes is taken as the parent node, and the corresponding child nodes are split off to store the data elements of the data in the data stream.

[0041] The data processing method provided in this application, for any data in the data stream, starts from the root node of the data representation tree and performs node recursive processing to determine the child node with the smallest distance for each recursively detected node, until the node with the smallest distance and no child nodes is obtained, which is used as the target node for storing the data element of the target data, and updates the variance of the target node. If the variance of the target node is greater than the corresponding variance threshold, the target node is used as the parent node and the corresponding child nodes are split off. Thus, based on distance, the node with the smallest distance in each level of the data representation tree is determined until the target node with the smallest distance is found. The smaller the distance, the more similar the data is to the features of the corresponding node. This allows data with similar features to be recursively sent to the child nodes corresponding to the same recursive detection node, which better reflects the relationship between the data. Furthermore, the smaller the variance, the more concentrated the data distribution; the larger the variance, the more dispersed the data distribution. Using variance as the basis for node splitting, when the data distribution is relatively dispersed, node splitting is triggered for the storage of subsequent data in the data stream. This ensures that the data elements in each node meet the data distribution requirements, better reflecting the data distribution characteristics. Moreover, the data elements stored in the same node are data elements with similar features, which better reflects the relationship between the data.

[0042] The data processing method provided in this application, such as Figure 2As shown, when performing corresponding data representation processing on the target data in the data stream that needs data representation processing, it is determined whether a data representation tree exists. If it does not exist, a data representation tree is constructed, and a node A included in the data representation tree is used as the root node. The data elements of the target data are stored in this node, and the node information of this node is updated. The node information includes the data count, mean, and variance.

[0043] Specifically, the data elements of the target data can be the target data itself, or at least one of the following: feature information, attribute information, and category information of the target data.

[0044] Furthermore, if the variance of the root node is greater than or equal to the corresponding variance threshold, the root node is used as the parent node, and corresponding child nodes, such as B1 and B2, are split off to store data elements in the data stream.

[0045] If the variance of the root node is less than the corresponding variance threshold, no split is performed.

[0046] After processing the next target data in the data stream, recursive node processing is performed starting from the root node of the data representation tree. First, the root node A is used as the first recursive detection node to determine whether the root node has child nodes. If the root node has no corresponding child nodes, the root node is used as the target node corresponding to the target data. The data elements of the target data are stored in the target node, and the node information of the target node is updated. Then, the variance of the target node is used to determine whether the root node needs to split the corresponding child nodes.

[0047] If the root node has corresponding child nodes B1 and B2, calculate the distance between the target data and the corresponding child node of the root node, and select the child node with the smallest distance, such as B1, as the next recursive detection node. If B1 has no corresponding child node, then B1 is taken as the target node corresponding to the target data, the data elements of the target data are stored in the target node B1, and the node information of the target node B1 is updated. Based on the variance of the target node B1, determine whether the target node B1 needs to split its corresponding child node.

[0048] This process continues until B1 splits into corresponding child nodes C1, C2, and C3. Then, the next target data in the data stream is processed. Starting from the root node of the data representation tree, node recursion is performed. If the root node has corresponding child nodes B1 and B2, the distance between the target data and B1 and B2 is determined. The node with the smallest distance is selected as the next recursive detection node. For example, if the node with the smallest distance is determined to be B1, then it is determined whether B1 has corresponding child nodes. As shown in the figure, if B1 has corresponding child nodes C1, C2, and C3, the distance between the target data and C1, C2, and C3 is determined. The node with the smallest distance, such as C2, is selected as the next recursive detection node. If C2 has no corresponding child nodes, then C2 is used as the target node. Data elements of the target data are stored, and the node information of C2 is updated. The variance of C2 is used to determine whether C2 needs to split into corresponding child nodes. For example, if the node with the smallest distance is determined to be B2, as shown in the figure, and B2 has no corresponding child nodes, then B2 is taken as the target node. The data elements of the target data are stored and the node information of B2 is updated. Based on the variance of B2, it is determined whether B2 needs to split its corresponding child nodes. This process continues until the variance of B2 is greater than or equal to the corresponding variance threshold. Then, with B2 as the parent node, the corresponding child nodes C4, C5, and C6 are split. Similarly, when the variance of C1 is greater than or equal to the corresponding variance threshold, with C1 as the parent node, the corresponding child nodes D1 and D2 are split, and the target data processing in the data stream continues.

[0049] Thus, each time the target data in the data stream is processed, the node with the smallest distance to the target data in each layer of nodes is determined hierarchically and recursively. Finally, the node with the smallest distance and no corresponding child node is determined as the target node. The data elements of the target data are stored in the target node, and the node information of the corresponding target node is updated. The variance determines whether to split the corresponding child node, and the split is hierarchical. As the data in the data stream increases, the data representation tree has more layers and greater depth.

[0050] In the implementation of this application, the data representation tree is a multi-way tree structure, that is, a multi-way tree is used as the information organization structure.

[0051] Multi-branch tree structures can meet the information representation needs of streaming data. Furthermore, dynamically splitting data representation tree structures can dynamically adjust the tree's depth and number of branches based on node variance. This allows the tree's nodes to dynamically split according to the data distribution variance, making the tree's distribution shape closer to the data distribution characteristics. Hierarchical recursion is performed based on distance, determining the node with the smallest distance to store data elements with similar characteristics under the same branch or even the same node. This allows the data representation tree structure to better reflect the relationships between data. Even with extremely large data volumes or high dimensionality, dynamic node splitting based on data variance can be used to represent new data and avoid redundant data storage.

[0052] In the implementation method of this application, such as Figure 3 As shown, calculating the distance between the target data and each child node corresponding to the recursive detection node includes the following steps.

[0053] S210, determine the current mean of each child node corresponding to the recursive detection node.

[0054] For example, determine the updated mean of each child node corresponding to the recursive detection node after the last data element storage, which is also the current mean.

[0055] It should be noted that if a node in the data representation tree does not store data, the mean of that node is the initial mean. That is, the current mean (i.e., the initial mean) of the nodes in the data representation tree that do not store data elements is empty, or the current mean of each node is generated based on the clustering method or the random generation method.

[0056] Specifically, the clustering method can be based on K-Means clustering for mean initialization.

[0057] Furthermore, in another implementation, the mean of nodes in the data representation tree that do not store data elements can also be determined based on the mean of their corresponding parent nodes. This is because the mean can reflect the characteristic information of the data. Determining the initial mean of the child nodes based on the mean of the parent nodes ensures that the data elements stored in the parent and child nodes under the same branch are data with characteristic associations, which can better reflect the distribution characteristics and relationships of the data.

[0058] Furthermore, the distance between the target data and each child node corresponding to the recursive detection node is calculated.

[0059] In the implementation of this application, the distance between the target data and each child node corresponding to the recursive detection node is used to represent the feature similarity between the target data and each child node corresponding to the recursive detection node. The smaller the distance, the higher the feature similarity.

[0060] S220, calculate the distance between the target data and each child node corresponding to the recursive detection node based on the current average value of the target data and each child node corresponding to the recursive detection node.

[0061] For example, based on the current average values ​​of the target data and the child nodes corresponding to the recursive detection node, the distance between the target data and the child nodes corresponding to the recursive detection node is calculated to determine the feature similarity between the target data and the child nodes corresponding to the recursive detection node.

[0062] In this implementation, for each child node corresponding to a recursive detection node at each level, the distance between the target data and each child node corresponding to the recursive detection node is calculated to determine the node with the highest feature similarity to the target data in each level, until the node with the highest feature similarity among the nodes without child nodes (i.e., leaf nodes) is determined as the target node. Thus, for a given target data, each level of nodes is determined as a node with feature similarity, and this process is repeated recursively, allowing similar data elements to be stored in the same branch or even the same node. This better reflects the feature similarity of data elements corresponding to the same node, and thus better reflects the data distribution relationship of data elements corresponding to the same branch node.

[0063] Specifically, the distance between the target data and the mean of each leaf node can be Euclidean distance, Manhattan distance, or other distance metrics.

[0064] Furthermore, in the implementation of this application, the current variance (i.e., the initial variance) of each node in the data representation tree that does not store data is empty, or the current variance of each node in the data representation tree that does not store data is generated based on a clustering method or a random generation method.

[0065] Alternatively, the variance of nodes in the data representation tree that do not store data elements can be determined based on the variance of their corresponding parent nodes. Therefore, parent nodes and child nodes are related. The initial variance of child nodes is determined based on the variance of the parent nodes, so that the dispersion of the data elements stored in the child nodes under the same branch can also reflect the dispersion of the data elements stored in the parent nodes, thus better reflecting the distribution characteristics of the data.

[0066] In the implementation of this application, the node information of each node in the data representation tree includes the data count, mean, and variance.

[0067] Among them, such as Figure 4 As shown, updating the node information of the target node includes the following steps.

[0068] S241, Increment the data count of the target node by one to update the data count of the target node.

[0069] For example, the number of data items in each node is counted based on a node data count counter. Each time a node stores a piece of data, the data count is incremented by one to obtain the latest data count for each node.

[0070] Furthermore, update the mean of the target node.

[0071] S242, determine the current mean of the target node, and determine the target mean of the target node based on the target data, the current mean, and the number of updated data, and update the mean of the target node to the target mean.

[0072] For example, the mean value updated after the last time the target node stored data is determined, which is the current mean value, is determined based on the target data, the current mean value, and the number of data after this update, so as to update the mean value of the target node to the latest mean value.

[0073] In this implementation, the target mean of the target node is determined based on the target data, the current mean, and the updated data count, including obtaining the target mean of the target node through the following methods:

[0074]

[0075] in, The target mean of the target node. This represents the current mean of the target node. The data vector of the target data. Count the number of data items updated for the target node.

[0076] Furthermore, update the variance of the target node.

[0077] S243, determine the current variance of the target node, and determine the target variance of the target node based on the current variance of the target node, the target data, the current mean of the target node, the target mean of the target node, and the number of data after the target node is updated, and update the variance of the target node to the target variance.

[0078] For example, the variance of the target node after the last storage of data elements is determined, which is the current variance. Based on the current variance of the target node, the target data, the current mean of the target node, the updated target mean, and the number of updated data elements of the target node, the target variance of the data elements of the target node after the last storage of target data is determined, so as to update the variance of the target node to the target variance.

[0079] In this implementation, the target variance of the target node is determined based on the current variance of the target node, the target data, the current mean of the target node, the target mean of the target node, and the count of the updated data of the target node. This is achieved by obtaining the target variance of the target node through the following methods:

[0080]

[0081] in, Let V be the target variance of the target node. Let be the current variance of the target node. The target mean of the target node. This represents the current mean of the target node. For target data, Count the number of data items updated for the target node.

[0082] Furthermore, based on the mean and variance of the streaming update as the basis for node splitting, when the variance of a node reaches a preset variance threshold, it automatically splits into multiple child nodes. Thus, when new data arrives, the closest child node is selected for recursive processing based on the distance between the new data and the child nodes at each level to obtain the nearest leaf node, which stores the data elements of the target data, and continues to update the data count, mean, and variance of that node. As the data in the data stream continues to increase, the tree structure will expand layer by layer, and the node splitting is controlled by the variance threshold to achieve a progressive construction of granularity from coarse to fine. The more data there is, the finer the granularity of the data representation tree becomes, which can better reflect the distribution characteristics of the data and avoid overfitting or underfitting problems caused by a fixed number of branches and a fixed node structure.

[0083] Furthermore, for nodes without child nodes in the data representation tree, if the variance of the node is greater than a preset variance threshold, it indicates that the data elements in that node are becoming increasingly discrete. Even if new data is stored in that node, the correlation between the data is weak. Therefore, this node is used as the parent node and splits into multiple child nodes. When new data is received, a new target node is determined based on the distance between the new data and each node without a corresponding child node. The new data is then stored in the new target node. This process is repeated to achieve streaming data storage, ensuring that the data elements in each node are data that meet the aggregation requirements and better reflect the distribution characteristics of the data.

[0084] In the implementation of this application, the number of child node splits is determined based on the attribute information of the data representation tree. The attribute information includes at least one of the following: dimension, data distribution, and data distribution requirements.

[0085] For example, the splitting of data can be dynamically determined based on at least one of the following: the distribution dimension of the data corresponding to the data representation tree, the data distribution of the data representation tree, and the data distribution requirements.

[0086] Furthermore, the aforementioned node without child nodes refers to the root node when the data representation tree only includes the root node, and to the leaf node when the data representation tree includes the root node and other nodes based on the root node.

[0087] The data processing method provided in this application is a progressive representation method for data information. It is mainly used to represent and organize the relationships between data based on a multi-branch tree structure to achieve data splitting or clustering. In other words, the data processing method provided in this application is a method for dynamically and progressively constructing the information representation structure of data. When new data x arrives in the data stream, node recursive processing begins from the root node. If it is determined that the recursive detection node has no child nodes, then the recursive detection node is taken as the target node corresponding to the target data. The data elements of the new data are stored in the target node, and the corresponding child nodes are split based on the node information of the target node and the variance of the target node. If it is determined that the recursive detection node has child nodes, the distance between the target data and each child node corresponding to the recursive detection node is calculated. The child node with the smallest distance is selected as the next recursive detection node, until the recursive detection node with the smallest distance and no child nodes is determined as the target detection node. Then, the corresponding data elements are stored, the node information is updated, and the node splitting judgment is performed. The data processing method based on this application can update the data representation tree structure in real time without waiting for all data to be collected, making it suitable for high-throughput data stream scenarios, such as processing sensor data, log data, and financial transaction data streams. When the data volume is small, the data representation tree is shallow, reflecting the coarse-grained distribution characteristics of the data. As data accumulates in the data stream, the data representation tree has more and more levels, more and more child nodes, and more and more branches, better reflecting the fine-grained distribution characteristics of the data. Furthermore, by triggering more nodes to split through a variance threshold, a deeper distribution tree structure is gradually formed, achieving a progressive construction from coarse to fine granularity. This avoids the overfitting or underfitting problems caused by a fixed structure. Compared with traditional binary trees, the depth and number of branches of this tree can adaptively expand according to the dispersion of the data distribution (i.e., variance), making the shape of the data representation tree closer to the data distribution characteristics. By using streaming mean and variance calculation, real-time statistics and updates can be performed without storing all the data, enabling real-time acquisition of the data distribution characteristics of each node. The data representation tree structure expands on demand, avoiding redundant storage. The data processing method provided in this application can serve as the basic structure for tasks such as real-time clustering, data compression, pattern recognition, and information retrieval, and is applicable to various data types such as numerical and vector types.

[0088] The data processing method provided in this application is applied to computer equipment.

[0089] Please see Figure 5 , Figure 5 The diagram shown is a structural schematic of a computer device provided in an embodiment of this application. Figure 5 As shown, the computer device may include: transceiver 121, processor 122, and memory 123.

[0090] Processor 122 executes computer execution instructions stored in memory, causing processor 122 to perform the technical solution of the data processing method in the above embodiments. Processor 122 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital data processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0091] The memory 123 is connected to the processor 122 via the system bus and completes communication between them. The memory 123 is used to store computer program instructions.

[0092] For example, and not as a limitation, memory 123 may include a hard disk drive (HDD), a floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or a universal serial bus (USB) drive, or a combination of two or more of these. Where appropriate, memory 123 may include removable or non-removable (or fixed) media. Where appropriate, memory 123 may be internal or external to the integrated gateway device. In a particular embodiment, memory 123 is non-volatile solid-state memory. In a particular embodiment, memory 123 includes read-only memory (ROM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable read-only ROM (PROM), an erasable programmable read-only ROM (EPROM), an electrically erasable programmable read-only ROM (EEPROM), an electrically alterable read-only ROM (EAROM), or flash memory, or a combination of two or more of these. Transceiver 121 can be used to obtain the task to be run and its configuration information.

[0093] The system bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The system bus can be divided into address bus, data bus, control bus, etc. For ease of representation, only one thick line is used in the diagram, but this does not indicate that there is only one bus or one type of bus. Transceivers are used to enable communication between database access devices and other computers (e.g., clients, read-write libraries, and read-only libraries). Memory may include random access memory (RAM) and may also include non-volatile memory.

[0094] The computer equipment provided in this application includes devices such as mobile phones, computers, vehicles, and servers.

[0095] This application also provides a chip for executing instructions, which is used to execute the data processing method described in the above embodiments.

[0096] This application also provides a computer-readable storage medium storing computer instructions that, when executed on a processor of a computer device, cause the processor of the computer device to perform the data processing method described in the above embodiments.

[0097] In some possible implementations, various aspects of the methods provided in this application may also be implemented as a program product, which includes program code. When the program product is run on the processor of a computer device, the program code is used to cause the processor of the computer device to perform the steps of the methods in the various exemplary implementations of this application described above. For example, the computer device may perform the data processing method described in the embodiments of this application.

[0098] The program product may take the form of any combination of one or more readable media. A readable medium may be a readable data medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CDROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0099] This application also provides a computer program product, which includes a computer program stored in a computer-readable storage medium. At least one processor can read the computer program from the computer-readable storage medium, and when the at least one processor executes the computer program, it can implement the technical solution of the data processing method in the above embodiments.

[0100] It should be noted that, in addition to the specific embodiments described above, those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. Although the description of this application is presented in conjunction with preferred embodiments, this does not mean that the features of this application are limited to this implementation. On the contrary, the purpose of describing the application in conjunction with the implementation is to cover other options or modifications that may be derived from this application. To provide a thorough understanding of this application, many specific details are included in the above description, and this application may also be implemented without using these details. Furthermore, to avoid confusion or obscuring the focus of this application, some specific details will be omitted in the description. It should be noted that, unless otherwise specified, the embodiments and features in the embodiments of this application can be combined with each other.

[0101] It should be noted that in this specification, similar reference numerals and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0102] It should be noted that the terms "first" and "second" are used only to distinguish descriptions and should not be interpreted as indicating or implying relative importance.

[0103] It should be noted that some structural or methodological features may be shown in the accompanying drawings in a specific arrangement and / or order. However, it should be understood that such a specific arrangement and / or order may not be necessary. Rather, in some embodiments, these features may be arranged in a manner and / or order different from that shown in the illustrative drawings. Furthermore, including structural or methodological features in a particular figure does not imply that such features are required in all embodiments, and in some embodiments, these features may be omitted or may be combined with other features.

[0104] Although this application has been illustrated and described with reference to certain preferred embodiments, those skilled in the art should understand that the above description is a further detailed explanation of the application in conjunction with specific implementations, and should not be construed as limiting the specific implementation of the application to these descriptions. Those skilled in the art can make various changes in form and detail, including some simple deductions or substitutions, without departing from the spirit and scope of this application.

Claims

1. A data processing method, characterized in that, The method includes: Determine the data representation tree corresponding to the target data, where the target data is any data in the data stream that needs to be processed for data representation. Starting from the root node of the data representation tree, node recursion is performed. The root node is used as the first recursive detection node. It is determined whether the recursive detection node has child nodes. If the recursive detection node has child nodes, the distance between the target data and each child node corresponding to the recursive detection node is calculated, and the child node with the smallest distance is selected as the next recursive detection node. If the recursive detection node has no child nodes, the recursive detection node is used as the target node corresponding to the target data. The data elements of the target data are stored in the target node, and the node information of the target node is updated. The node information includes variance. Furthermore, if the variance of a node without child nodes in the data representation tree is greater than or equal to the corresponding variance threshold, the node without child nodes is used as the parent node, and corresponding child nodes are split off to store the data elements of the data in the data stream.

2. The data processing method according to claim 1, characterized in that, Calculating the distance between the target data and each child node corresponding to the recursive detection node includes: Determine the current average value of each child node corresponding to the recursive detection node; Based on the target data and the current average value of each child node corresponding to the recursive detection node, calculate the distance between the target data and each child node corresponding to the recursive detection node.

3. The data processing method according to claim 2, characterized in that, The node information also includes a data count and an average value. Updating the node information of the target node includes: Increment the data count of the target node by one to update the data count of the target node; Determine the current mean of the target node, and based on the target data, the current mean, and the updated data count, determine the target mean of the target node, and update the mean of the target node to the target mean.

4. The data processing method according to claim 3, characterized in that, The target mean of the target node is determined based on the target data, the current mean, and the updated data count, including obtaining the target mean of the target node in the following manner: in, Let be the target mean of the target node. The current mean of the target node. For the target data, The number of data items updated for the target node is counted.

5. The data processing method according to claim 4, characterized in that, Updating the node information of the target node also includes: Determine the current variance of the target node; The target variance of the target node is determined based on the current variance of the target node, the target data, the current mean of the target node, the target mean of the target node, and the number of updated data points of the target node. Update the variance of the target node to the target variance.

6. The data processing method according to claim 5, characterized in that, The target variance of the target node is determined based on the current variance of the target node, the target data, the current mean of the target node, the target mean of the target node, and the updated data count of the target node. This determination includes obtaining the target variance of the target node through the following methods: in, Let V be the target variance of the target node. Let V be the current variance of the target node. Let be the target mean of the target node. The current mean of the target node. For the target data, The number of data items updated for the target node is counted.

7. The data processing method according to claim 6, characterized in that, The current mean and current variance of nodes in the data representation tree that do not store data elements are empty, or the current mean and current variance of nodes in the data representation tree that do not store data elements are generated based on clustering methods or random generation methods.

8. The data processing method according to any one of claims 1-6, characterized in that, The number of splits of the child node is determined based on the attribute information of the data representation tree, which includes at least one of the following: dimension, data distribution, and data distribution requirements.

9. The data processing method according to any one of claims 1-6, characterized in that, The data representation tree is a multi-branch tree structure.

10. A computer device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes the computer execution instructions stored in the memory to cause the computer device to perform the data processing method as described in any one of claims 1-9.

Images