A data processing method and apparatus

Abstract

This application provides a data processing method and apparatus for mining data based on the relationships between elements to obtain patterns with stronger correlations. The method includes: acquiring a first dataset containing multiple elements, where each element can be used as a central element, and each central element is associated with at least one leaf element; acquiring the encoding length of the at least one central element to obtain first encoded data for each central element, wherein the encoding length of the at least one central element is related to the frequency of occurrence of the at least one central element in the first dataset; acquiring the encoding length of each leaf element associated with each central element based on the first encoded data to obtain second encoded data for each leaf element; and updating the first dataset based on the first encoded data of each central element and the second encoded data of each leaf element to obtain a second dataset.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and more particularly to a data processing method and apparatus. Background Technology

[0002] Association rule algorithms, based on customer transaction databases, are used to solve the problem of relationships between itemsets in large transactional or relational datasets. The purpose of association rule analysis is to discover hidden connections between data points in large datasets; the discovered patterns are typically represented as association rules or frequent itemsets. Besides transactional data, graph data is also widely used, for example, in fields such as chemistry, biology, and social networks.

[0003] Some commonly used data mining methods can uncover relationships between data points based on frequent itemsets or the minimum description length (MDL) algorithm. However, these methods often yield incomplete information and may fail to reveal the relationships between elements. Therefore, how to mine more comprehensive data has become a pressing issue. Summary of the Invention

[0004] This application provides a data processing method and apparatus for mining data based on the relationships between elements to obtain patterns with stronger correlations.

[0005] In view of the above, in a first aspect, this application provides a data processing method, comprising: firstly, obtaining a first dataset, the first dataset including multiple elements, wherein each of the multiple elements can be used as a central element, and each central element is associated with at least one leaf element; obtaining the encoding length of at least one central element to obtain first encoded data of each central element, wherein the encoding length of at least one central element is related to the number of times the at least one central element appears in the first dataset; obtaining the encoding length of each leaf element associated with each central element based on the first encoded data to obtain second encoded data of each leaf element, wherein the encoding length of each leaf element associated with each central element is related to the number of times each central element and each leaf element associated with each central element appears in the first dataset; and updating the first dataset based on the first encoded data of each central element and the second encoded data of each leaf element to obtain a second dataset.

[0006] Therefore, in this embodiment of the application, data mining can be performed based on central elements and leaf elements, resulting in central and leaf elements with strong correlations. This allows for the discovery of stronger relationships between elements. Even with complex data, highly correlated patterns can be mined, making it widely applicable across various scenarios and demonstrating strong generalization ability.

[0007] In one possible implementation, the aforementioned process of updating the first dataset based on the first encoded data of each central element and the second encoded data of each leaf element to obtain the second dataset may include: selecting at least two leaf elements from the first dataset that are associated with the same central element and merging them based on the first encoded data of each central element and the second encoded data of each leaf element to obtain the second dataset.

[0008] Therefore, in this embodiment, leaf element pairs with the same central element can be merged to reduce data redundancy in the dataset, uncover patterns with stronger correlation, and achieve more effective data mining.

[0009] In one possible implementation, the aforementioned process of selecting at least two leaf elements associated with the same central element from the first dataset and merging them to obtain the second dataset, based on the first encoded data of each central element and the second encoded data of each leaf element, may include: using the first encoded data of each central element and the second encoded data of each leaf element to calculate the compression contribution rate corresponding to each set of leaf element pairs in a plurality of sets of leaf element pairs, wherein the compression contribution rate is used to represent the change in description length when each set of leaf element pairs is merged, wherein each set of leaf element pairs includes at least two leaf elements in the first dataset that have the same central element; and selecting at least one set of leaf element pairs from the plurality of sets of leaf element pairs and merging them to obtain the second dataset.

[0010] Therefore, in the embodiments of this application, the change in the description length of each group of leaf element pairs before and after merging can be represented by the compression contribution rate. This description length is usually related to the encoding length of each group of leaf element pairs. Therefore, leaf elements with high compression contribution rates can be selected from multiple groups of leaf element pairs for merging, thereby mining patterns with strong correlations to a greater extent.

[0011] In one possible implementation, the aforementioned method further includes: if the compression contribution rate of multiple sets of leaf element pairs does not meet the preset condition, then the second dataset is used as the new first dataset, and the center element and leaf element in the new first dataset are re-encoded respectively; if the compression contribution rate of multiple sets of leaf element pairs meets the preset condition, then the updating of the first dataset is stopped.

[0012] Therefore, in this embodiment of the application, the dataset can be iteratively mined until a dataset with low redundancy is finally obtained, thereby preserving the strongly correlated patterns to the greatest extent and mining the even more correlated patterns.

[0013] In one possible implementation, the encoding length of the first encoded data is negatively correlated with the frequency of each central element's occurrence in the first dataset. Therefore, the probability of a central element's occurrence can be represented by its encoding length, thereby identifying central elements with higher occurrence probabilities.

[0014] In one possible implementation, the ratio between the first count and the second count is negatively correlated with the encoding length of the second encoded data for each leaf element. The first count is the number of times each leaf element in the first dataset appears simultaneously with its associated central element, and the second count is the number of times the central element associated with each leaf element in the first dataset appears alone.

[0015] Therefore, in this embodiment of the application, the probability of each center element and its corresponding leaf element appearing at the same time can be represented by the encoding length, thereby mining each center element and its corresponding leaf element with a higher probability of occurrence, i.e., center-leaf element pairs.

[0016] In one possible implementation, the aforementioned acquisition of the first dataset may include: acquiring input data, which includes multiple elements and the relationships between elements; constructing a flipped database by using each element as a central element and the elements associated with each central element as leaf elements, the flipped database including at least three columns, which include columns formed by the central element, leaf elements, and the positions of the central element in the input data, and the first dataset includes the flipped database.

[0017] Therefore, in this embodiment, each element in the input data can be used as the central element, and the elements associated with the central element can be used as leaf elements. A flipped database is constructed, which includes the central element, leaf elements, and the position of the central element in the input data. Thus, the position of each element and combination of elements in the input data can be displayed more comprehensively through the flipped database, reducing data redundancy.

[0018] In one possible implementation, the multiple elements in the first dataset include alarm information for each of the multiple devices, or the multiple elements in the first dataset include multiple alarm information for a single device. Correspondingly, the second dataset may include the correlation between multiple alarms, which can be multiple alarms from a single device or multiple alarms from multiple devices. Therefore, the embodiments of this application can be applied to device alarm scenarios to identify related important alarms and filter or collapse non-important alarms.

[0019] In one possible implementation, the method may further include: acquiring a first alarm generated on a first device, acquiring a second alarm associated with the first alarm from a second dataset, thereby predicting a second alarm that is about to be generated or is being generated on the first device. Therefore, in this embodiment, when a device generates an alarm, the association relationship can be mined to predict whether an alarm is about to be generated or has already been generated, thereby enabling timely response to device alarms and improving device security.

[0020] In one possible implementation, multiple elements in the first dataset include tags for multiple users; alternatively, multiple elements in the first dataset include multiple tags for a single user, and the second dataset may include relationships between these multiple tags, which could include multiple tags for a single user or multiple tags for multiple users. This allows for the discovery of relationships between user characteristics, thereby enabling corresponding recommendation functions.

[0021] In one possible implementation, the above method may further include: obtaining a first tag of a first user, and then obtaining a second tag associated with the first tag from a second dataset, so that user recommendations can be made subsequently based on the first and second tags. Therefore, in this embodiment, tag associations with high correlation strength can be mined, and user recommendations can be made based on these associations, thereby enabling accurate and timely recommendations to users and improving user experience.

[0022] Secondly, embodiments of this application provide a data processing apparatus that has the function of implementing the data processing method described in the first aspect. This function can be implemented in hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the aforementioned function.

[0023] Thirdly, embodiments of this application provide a data processing apparatus, including a processor and a memory, wherein the processor and the memory are interconnected via a circuit, and the processor calls program code in the memory to execute processing-related functions in the data processing method shown in any of the first aspects above. Optionally, the data processing apparatus may be a chip.

[0024] Fourthly, embodiments of this application provide a digital processing chip or chip, the chip including a processing unit and a communication interface, the processing unit obtaining program instructions through the communication interface, the program instructions being executed by the processing unit, the processing unit being used to perform processing-related functions as described in any of the optional embodiments of the first or second aspect above.

[0025] Fifthly, embodiments of this application provide a computer-readable storage medium including instructions that, when executed on a computer, cause the computer to perform the method in any of the optional embodiments of the first or second aspect described above.

[0026] Sixthly, embodiments of this application provide a computer program product comprising a computer program / instructions, which, when executed by a processor, causes the processor to perform the method in any of the optional embodiments of the first or second aspect described above. Attached Figure Description

[0027] Figure 1 The system architecture diagram provided for this application;

[0028] Figure 2 A schematic diagram of a data format provided in this application;

[0029] Figure 3 A flowchart illustrating a data processing method provided in this application;

[0030] Figure 4 A schematic diagram illustrating another data format provided in this application;

[0031] Figure 5 A flowchart illustrating another data processing method provided in this application;

[0032] Figure 6 A schematic diagram illustrating another data format provided in this application;

[0033] Figure 7 A schematic diagram illustrating another data format provided in this application;

[0034] Figure 8 A schematic diagram of the central element encoding table provided in this application;

[0035] Figure 9 A schematic diagram of the leaf element encoding table provided in this application;

[0036] Figure 10 A schematic diagram of a pattern update process provided for this application;

[0037] Figure 11 A schematic diagram illustrating a pattern merging method provided in this application;

[0038] Figure 12 A schematic diagram illustrating a database update method provided in this application;

[0039] Figure 13 This application provides an illustration of an application scenario.

[0040] Figure 14 This application provides another application scenario illustration;

[0041] Figure 15 This application provides another application scenario illustration;

[0042] Figure 16 This application provides another application scenario illustration;

[0043] Figure 17 A schematic diagram of the structure of a data processing device provided in this application;

[0044] Figure 18 A schematic diagram of another data processing device provided in this application. Detailed Implementation

[0045] The technical solutions of the embodiments of this application will be described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0046] First, the data processing method provided in this application can be deployed on various servers or terminals and other devices to mine and analyze the relationships between various elements in the data collected by the server or terminal.

[0047] For example, such as Figure 1 As shown, the method provided in this application can be executed by a server that can establish connections with multiple clients, specifically through wired or wireless communication connections.

[0048] The communication network can be a wide area network (WAN), a local area network (LAN), a point-to-point connection, or any combination thereof. Specifically, the communication network can include a wireless network, a wired network, or a combination of wireless and wired networks. The wireless network includes, but is not limited to, any one or more combinations of: 5th-Generation (5G) systems, Long Term Evolution (LTE) systems, Global System for Mobile Communication (GSM) or Code Division Multiple Access (CDMA) networks, Wideband Code Division Multiple Access (WCDMA) networks, Wireless Fidelity (WiFi), Bluetooth, Zigbee, Radio Frequency Identification (RFID), Long Range (Lora) wireless communication, and Near Field Communication (NFC). The wired network can include fiber optic communication networks or networks composed of coaxial cables.

[0049] The server can receive information from the client and analyze it to obtain the individual elements and relationships between them. The server can also perform correlation analysis on information it has collected; the specific deployment depends on the actual application scenario, and this application does not impose any limitations on this.

[0050] Of course, the terminal can also analyze the information it collects to obtain the relationships between the various elements in the information it collects.

[0051] Data requiring correlation analysis can typically be categorized into several types, such as transactional data or graphical data. For example, transactional data can be represented as shown in Table 1.

[0052] TID Product List 1 A,B,C 2 A,B 3 A,B,C

[0053] Table 1

[0054] In Table 1, each row corresponds to a transaction, and each transaction has a unique identifier (TID). ABC represent items in the dataset. A subset of these items is called an itemset. An itemset can be empty, meaning it doesn't contain any items. If an itemset contains k items, it is called a k-itemset. Orders 1, 2, and 3, as shown in Table 1, are called transactions. The number of times an itemset appears in all transactions is called the support count of the itemset. For example, the support count for A is 3, for B it is 3, and for C it is 2. The purpose of association rule analysis is to discover the relationships between elements in a dataset. The discovered patterns are usually represented by association rules or frequent itemsets.

[0055] For example, graphical data is also widely used in fields such as chemistry, biology, monitoring, and social networks. One example of graphical data is... Figure 2 As shown, the graph data includes nodes (v1, v2, v3, v4, v5), the relationships between nodes, and the corresponding attributes (a, b, c) for each node. These attributes can also be called elements or attribute elements. A graph dataset may contain one or more graphs. Analogous to frequent item mining in transaction data, node attribute relationship mining has wide-ranging applications, such as mining alarm relationships in network devices and user feature analysis in recommendation systems.

[0056] Association analysis aims to utilize pattern mining, which can discover connections such as association rules or frequent itemsets. These connections or correlations can be used to better understand the data and provide relevant information for subsequent decision-making.

[0057] Frequent itemset mining is a primary method in association analysis, with the main challenge being the sheer number of patterns discovered. To compress the massive number of frequent patterns generated during mining while maintaining high-quality patterns, compressed sets or approximate sets of frequent patterns can be mined. Secondly, pattern mining algorithms typically require setting multiple parameters to select patterns. However, finding suitable values ​​for these parameters is time-consuming and often unintuitive.

[0058] For example, in frequent itemset mining, a parameter called minimum support must be set. Choosing the appropriate minimum support is very difficult because it depends on the features of the dataset that are initially unknown to the user. Therefore, users often run the algorithm multiple times with different parameter values ​​to analyze and filter the obtained patterns. If the constraints defined by the parameter values ​​are too strict, many useful patterns will be missed, while if they are too loose, a large number of stray patterns may be found, making it difficult to sort through them.

[0059] Other parameter-free pattern mining methods, such as SLIM or Kprim, do not provide information on the strength of associations between patterns; they only analyze frequency information. They can only handle simple transaction datasets or mining subgraph structures, and cannot handle complex structures like attribute graphs or the mining of associations between attributes.

[0060] For example, in frequent item-based association mining, the frequent item mining algorithm first identifies the set of itemsets where each frequent item has a frequency at least less than a predetermined minimum support (min_sup). Then, association rules are derived from these frequent itemsets based on a selected confidence level (min_conf). However, the mined patterns (sets) cannot assess the importance of items within a pattern. Frequent item mining-based association analysis algorithms require pre-setting a minimum support count (min_sup). A large value results in a very low support threshold to ensure comprehensive rules, thus preventing inefficiency due to the large scale of rules. If min_sup is set too low, a large number of patterns are output, leading to pattern explosion; if min_sup is set too low, only a very small number of patterns are found. For graph data, the patterns output by frequent subgraph mining algorithms are frequent subgraphs, which cannot analyze the relationships between attributes.

[0061] For example, the basic idea of ​​the MDL-based algorithm is to design an encoding scheme and use a set of patterns as a dictionary to encode the original sequence. Since the required description length is smaller after encoding, this process can be regarded as compression of the original sequence information. By finding the set with the best compression capability as the result, a compact pattern result can be obtained. The pattern set selected using the MDL criterion can compress the information in the original sequence to the greatest extent. Commonly used MDL-based sequence compression pattern mining algorithms (Krimp, Slim) mine unordered patterns and do not consider the importance of elements in the pattern. When mining attribute patterns in graph data, the topological information of the graph data also needs to be discarded and transformed into simple transaction data, resulting in the loss of data association relationships. Therefore, the patterns mined based on MDL cannot indicate the strength of the association between patterns, but can only analyze the frequency. Furthermore, it can only mine simple transaction datasets or subgraph structures, and cannot mine complex structures such as attribute graphs or the association relationships between attributes.

[0062] Therefore, this application provides a data processing method that can mine the relationships and strengths of associations between elements, and discover corresponding association patterns that are highly interpretable and distinguishable.

[0063] First, for ease of understanding, some terms involved in the data processing method provided in this application will be explained.

[0064] Central element: Each element in the dataset can be called the central element.

[0065] Leaf element: The element in a dataset that is associated with the central element is called a leaf element. An element can be either the central element or a leaf element.

[0066] Pattern: A set of related center elements and leaf elements can be called a pattern.

[0067] Model: A model is a set of central elements and patterns that include those central elements.

[0068] The data processing methods provided in this application are described below.

[0069] See Figure 3 The following is a flowchart illustrating a data processing method provided in this application.

[0070] 301. Obtain the first dataset.

[0071] The first dataset may include various elements, with one or more of each element. This can be understood as having multiple nodes, each node having one or more elements, and nodes with relationships are connected, with the elements within the connected nodes being related.

[0072] Optionally, the process of obtaining the first dataset may include: obtaining initial input data, which includes multiple elements and the relationships between elements; using each element as a central element and the elements associated with each central element as leaf elements; constructing a flipped database, which includes at least three columns, including the central element, leaf elements, and columns formed by the positions of the central element in the input data; a row in the flipped database can be understood as a pattern; the first dataset includes the flipped database, or the first data can be the flipped database.

[0073] For example, in a device alarm scenario, the first dataset may include alarm information from multiple devices, such as alarm type, alarm root cause, or alarm identifier, which can associate devices that have alarms at the same time. Alternatively, the first dataset may include multiple alarm information from one device, which can associate the root causes of alarms that occur at the same time on that device.

[0074] For example, in a user recommendation scenario, the first dataset may include one or more tags for multiple users, such as the user's preferred music or movie genres, which can link users with the same tags together.

[0075] Typically, if this application is executed by a server, the server can receive information from one or more clients to obtain the first dataset. For example, a network maintenance center can be deployed on a server. When one or more devices malfunction, an alarm storm is triggered, and the network maintenance center will receive a massive number of warnings, including information on which devices in the network have malfunctioned and what kind of malfunctions have occurred.

[0076] Optionally, the process of obtaining the first dataset may specifically include, if the method provided in this application is executed by a server, the server may preprocess the received or collected input data, organize the various elements in the initial data, such as assigning each element the role of a central element, and assigning the elements associated with the central element the role of leaf elements, thereby obtaining the first dataset.

[0077] Specifically, elements in the initial input data can be assigned corresponding roles. For example, each element in the first dataset can be considered a central element, and the elements associated with the central element can be considered leaf elements. Typically, when an element is a central element, and it has associated elements, that central element can also be a leaf element of its associated elements; that is, each element can be both a central element and a leaf element.

[0078] For example, the initial input data can be like this: Figure 4 As shown, this first dataset can include multiple elements, each of which can be designated as a central element, and the elements associated with the central element can be designated as leaf elements. There is a substantial topological connection between the corresponding nodes of the central element and the leaf elements. For example, the central element might be a user's music preferences, and the leaf elements might be the music preferences of that user's friends. In some possible scenarios, a central element might only have one leaf attribute. This can be understood as the central element being the most important element in the dataset, while the leaf elements are those strongly correlated with the central element. For example... Figure 4 As shown in the figure, a, b, and c are the elements in the dataset. Taking the element in v1 as the center element as an example, v1 is associated with v2, v3, and v4 respectively, and can form center-leaf element pairs (a,a), (a,c), and (a,b). (a,a) means that element a is a leaf element of element a, (a,c) means that element c is a leaf element of element a, and (a,b) means that element b is a leaf element of element a.

[0079] Furthermore, for the convenience of those skilled in the art, the simultaneous occurrence of a central element and a leaf element can be referred to as a pattern, such as (a,b) can be referred to as a pattern, and (a,c) can be referred to as a pattern.

[0080] 302. Obtain the encoding length of at least one central element, and obtain the first encoded data corresponding to each central element.

[0081] There are several ways to obtain the encoding length of the central element, such as using Shannon entropy or information entropy.

[0082] Specifically, the encoding length of the first encoded data is related to the frequency of occurrence of the central element in the first dataset. For example, the encoding length of the first encoded data and the frequency of occurrence of the central element in the first dataset can be negatively correlated. For instance, Shannon entropy can be used to measure the encoding length of the central element, that is, the cumulative change distribution function of the source symbols is used to allocate codewords, as can be expressed as:

[0083]

[0084] Among them, f c F represents the frequency of occurrence of the central element, and F is the sum of the frequencies of occurrence of all central elements.

[0085] 303. Obtain the encoding length of each leaf element to get the second encoding data corresponding to each leaf element.

[0086] The calculation of the encoding length of leaf elements can be performed in various ways, such as using Shannon entropy, conditional entropy, or other methods. When calculating the encoding length of a leaf element, it can be based on the number of times the leaf element and its corresponding center element appear (referred to as the first occurrence count) and the number of times the corresponding center element appears (referred to as the second occurrence count). The ratio between the first and second occurrence counts is negatively correlated with the encoding length of the second encoded data. This can be understood as the ratio between the first occurrence count of a leaf element and its associated center element appearing together, and the second occurrence count of the associated center element appearing alone, which is negatively correlated with the encoding length of the second encoded data. For example, the encoding length of a leaf element can be calculated using conditional entropy, as shown below:

[0087]

[0088] Among them, f c f represents the frequency of occurrence of the central element. L This indicates the frequency at which a leaf element and its corresponding center element appear simultaneously.

[0089] Therefore, in this embodiment of the application, the encoding length of the first encoding data and the second encoding data can be used to reflect the association strength between each leaf element and its associated central element, so that the dataset can be compressed subsequently to obtain the final dataset of strongly associated elements.

[0090] It should be noted that after distinguishing between the center element and the leaf element, the center element can be encoded first, the leaf element can be encoded first, or both the center element and the leaf element can be encoded simultaneously. That is, step 302 can be executed first, step 303 can be executed first, or both steps 302 and 303 can be executed simultaneously. The specific method can be adjusted according to the actual application scenario, and this application does not limit it.

[0091] 304. Update the first dataset based on the first encoded data and the second encoded data to obtain the second dataset.

[0092] Specifically, after encoding the center element and leaf element, the encoded data of the center element and leaf element can be used to update the center element and leaf element in the first dataset respectively, thus obtaining the second dataset.

[0093] The second dataset includes one or more patterns. Optionally, when the second dataset includes multiple patterns, the pattern with the shortest description length can be selected as the final output pattern.

[0094] Therefore, in this embodiment, data mining can be performed based on central and leaf elements, yielding central and leaf elements with strong correlations, thus revealing stronger relationships between elements. Even with complex data, highly correlated patterns can be mined, making it widely applicable across various scenarios and demonstrating strong generalization ability. Furthermore, compared to association mining based on frequent items, the mining method provided in this application does not require calculating minimum support, thus avoiding the limitations of minimum support and enabling the discovery of more effective patterns. Moreover, compared to MDL-based association mining methods, the mining method provided in this application can uncover highly correlated patterns and can mine more complex data, demonstrating greater adaptability.

[0095] In one possible implementation, the process of updating the first dataset may include: merging leaf element pairs with the same central element selected from the first dataset based on the first coded data and the second coded data to obtain the second dataset. Therefore, in this embodiment, leaf element pairs with the same central element can be merged, thereby reducing data redundancy in the dataset, uncovering patterns with stronger correlations, and achieving more effective data mining.

[0096] Specifically, during the merging of leaf element pairs, the first and second encoded data can be used to predict the compression contribution rate of each pair. This compression contribution rate represents the change in description length when each pair of leaf elements is merged. Then, based on the compression contribution rate of each pair, leaf element pairs with the same central element are selected from multiple pairs and merged to obtain the second dataset. Specifically, this can be achieved by selecting leaf element pairs with a compression contribution rate greater than a preset value for merging, thereby uncovering more strongly correlated patterns.

[0097] More specifically, the encoding length of a model can include the encoding length of the center element and the encoding length of the leaf element in the model. For example, the encoding length of model a+(a,b) can be represented as the encoding length of the center element a and the encoding length of the leaf element (a,b).

[0098] Furthermore, if the compression contribution rate of multiple leaf element pairs does not meet the preset conditions, the second dataset can be used as the new first dataset, and step 301 can be iterated again until the compression contribution rate of multiple leaf element pairs in the final dataset meets the preset conditions.

[0099] Specifically, the preset condition could be that the compression contribution rate of the multiple sets of leaf elements is less than the first threshold, or the change in compression contribution rate between multiple iterations is less than the second threshold, etc., which can be adjusted according to the actual application scenario.

[0100] Therefore, in this embodiment of the application, the dataset can be iteratively mined until a dataset with low redundancy is finally obtained, thereby preserving the strongly correlated patterns to the greatest extent and mining the even more correlated patterns.

[0101] For example, if the first dataset includes multiple elements that include alarm information for each of the multiple devices, or if the first dataset includes multiple elements that include multiple alarm information for a single device, then the second dataset may include the relationships between multiple alarms. These multiple alarms may be multiple alarms in a single device or multiple alarms from multiple devices.

[0102] Optionally, a first alarm generated on the first device can be obtained, and a second alarm associated with the first alarm can be obtained from the second dataset, thereby predicting a second alarm that is about to be generated or is being generated on the first device. Therefore, in this embodiment of the application, when the device generates an alarm, the alarm that the device is about to generate or has already generated can be predicted based on the correlation obtained by mining, thereby responding to the device alarm in a timely manner and improving the security of the device.

[0103] For example, if the first dataset includes multiple elements that include tags for multiple users; or if the first dataset includes multiple elements that include multiple tags for one user, then the second dataset may include the relationships between multiple tags, which may include multiple tags for one user or multiple tags for multiple users.

[0104] Optionally, a first tag of the first user can be obtained, and a second tag associated with the first tag can be obtained from the second dataset. This allows for subsequent user recommendations based on the first and second tags. Therefore, in this embodiment, highly correlated tag relationships can be identified, and user recommendations can be made based on these relationships, resulting in accurate and timely recommendations and improved user experience.

[0105] The foregoing has described the flow of the data processing method provided in this application. To facilitate understanding, the data processing method provided in this application will be described in more detail below in conjunction with specific application scenarios.

[0106] For example, see Figure 5 The flowchart of another data processing method provided in this application can be divided into several steps, namely data preprocessing 501, obtaining the central element code table 502, obtaining the leaf element code table 503, and pattern update 504.

[0107] Data preprocessing: This can be understood as organizing the initial data into a dataset that includes multiple elements, or as flipping the database.

[0108] Obtain the central element code table: This involves taking each element in the dataset as the central element and calculating the code length to obtain the central element code table.

[0109] Obtain the leaf element code table: take the elements in the dataset associated with the center element as leaf elements, and calculate the encoding length of the leaf elements in the dataset to obtain the leaf element code table.

[0110] Pattern update: The dataset can be updated using the central element code table and the leaf element code table to obtain the updated dataset.

[0111] The steps described above will be explained in more detail below.

[0112] I. Data Preprocessing

[0113] After obtaining the initial input data, the central and leaf elements in the input data are organized.

[0114] The input data can be transactional data or graphical data; for example, a star chart, such as... Figure 6As shown, the star diagram includes multiple nodes, each with one or more elements. Nodes with relationships are interconnected, such as... Figure 7 As shown, each element in this star chart can be used as a central element, and leaf elements connected to each central element can be identified.

[0115] This can be understood as assigning two attributes to each vertex element of a star diagram: a central element and a leaf element. Associating the central and leaf elements with actual connections yields the basic star diagram, used to record the vertices containing the central elements. The associated element pairs can be merged and integrated into preprocessed data, which can be called an "inverted database" for ease of understanding.

[0116] For example, the data of the central element obtained after preprocessing can be shown in Table 2, where Sc represents the central element and the second column represents the position of the node where the central element appears, which can be used to calculate the number of times the central element appears.

[0117] <![CDATA[S c ]]> Postion a <![CDATA[v1,v2,v5]]> b <![CDATA[v4]]> c <![CDATA[v2,v3]]>

[0118] Table 2

[0119] For example, leaf elements can be integrated based on the central element. The resulting leaf elements after preprocessing are shown in Table 3.

[0120]

[0121] Table 3

[0122] Where Sc represents the central element S L Representing leaf elements, Sc and S in a row L This forms a pattern. The leaf elements obtained after preprocessing can also be understood as a flipped database (i.e., the first dataset). By flipping the database, we can represent the various elements in the input data and the relationships between them. We can summarize all possible star-shaped attribute structures without redundancy, and obtain the frequency or number of occurrences of each star structure without scanning the entire dataset. Furthermore, if a new pattern is generated, when merging the new pattern, we only need to take the intersection of the positions of the patterns to be merged, which allows for efficient pattern merging.

[0123] After preprocessing the input data to obtain the data corresponding to the center element and the data corresponding to the leaf element group, they can be encoded separately.

[0124] II. Obtaining the Central Element Code Table

[0125] The central element can be encoded in various ways, such as Huffman coding, arithmetic coding, or Shannon-Fanno coding, and the corresponding coding length can be obtained by using methods such as Shannon entropy or conditional entropy to obtain the central element coding table.

[0126] For example, the central element encoding table can be as follows: Figure 8 As shown, the central element encoding table can include at least two parts: the central element and the encoding length corresponding to each central element. For example, Huffman coding can be used for encoding, followed by Shannon entropy for calculating the encoding length, and the cumulative distribution function of the source symbols can be used to allocate codewords. The encoding length can be calculated as follows:

[0127]

[0128] Among them, f c F represents the frequency of occurrence of the central element, and F is the sum of the frequencies of occurrence of all central elements. The longer the final code length, the higher the frequency or number of times the central element appears.

[0129] III. Obtaining the Leaf Element Code Table

[0130] When encoding leaf elements, there are various ways to encode them, such as Huffman coding, and the encoding length can be calculated using encoding algorithms such as Shannon entropy, conditional entropy, or other encoding algorithms related to the frequency of pattern occurrence, to obtain the leaf element code table.

[0131] For example, leaf elements can be encoded using Huffman coding, arithmetic coding, or Shannon-Fanno coding, and the optimal code length can be calculated using conditional entropy. The code length can be calculated as follows:

[0132]

[0133] Among them, f c f represents the frequency of occurrence of the central element. L This represents the frequency of the co-occurrence of the center element and the leaf element associated with each leaf element. The resulting leaf element code table can be as follows: Figure 9 As shown, the ratio between the number of times a leaf element and its associated central element appear together and the number of times the central element in a leaf element pair appears alone is negatively correlated with the encoding length. That is, the larger the ratio between the number of times a leaf element and its associated central element appear together and the number of times the central element in a leaf element pair appears alone, the smaller the encoding length.

[0134] Therefore, the encoding length of leaf elements can also be used to reflect the probability of a associated leaf element appearing when a central element appears. For example, taking the leaf element (a, b) as an example, the ratio between the number of times a and b appear simultaneously and the number of times a appears is negatively correlated with the encoding length. The larger this ratio, the shorter the encoding length; the smaller the ratio, the longer the encoding length. Therefore, the encoding length can represent the probability of each leaf element appearing when the central element appears. This can be understood as representing the strength of the association between elements, thereby enabling the discovery of patterns with stronger associations in the dataset.

[0135] IV. Mode Update

[0136] Specifically, when updating the mode, both the flipped database and the dual-key table (i.e., the central element key table and the leaf element key table) can be updated.

[0137] In particular, when updating the flipped database and dual-code table, an MDL-based approach can also be used. The description length L can be represented as L(H) + L(D / H), where L(H) represents the encoding length of the model, and L(D / H) represents the encoding length of the dataset after representing it using the pattern.

[0138] Schema updates can also be divided into multiple steps, such as Figure 10 As shown, this can specifically include selecting alternative modes and updating modes.

[0139] For example, we can predict the compression contribution of each group of leaf attributes after merging, and then select the patterns with higher compression contribution from multiple groups of leaf element pairs as candidate patterns based on the compression contribution. We can then use the encoded data of the candidate patterns to update the corresponding patterns in the first dataset, thereby compressing the dataset and obtaining the compressed patterns.

[0140] The following describes the steps for selecting alternative modes and updating modes.

[0141] 1. Select alternative mode

[0142] It can calculate the contribution of each group of leaf element pairs to the compression of the dataset after merging, and then select one or more candidate patterns from multiple groups of leaf element pairs.

[0143] Specifically, the compression contribution (Gain) can be calculated as follows: ΔL = Description length of the pattern before update - Description length of the pattern after update, where the description length of the pattern is related to the encoding length of the pattern. For example, the encoding length of the pattern can be directly used as the description length, or the encoding length of the pattern can be converted into the description length.

[0144] Multiple patterns can be sorted according to their compression contribution value, and then a certain number of patterns with higher compression contribution values ​​can be selected as candidate patterns.

[0145] 2. Mode Update

[0146] After selecting one or more pairs of candidate patterns, the selected patterns can be merged, that is, the selected one or more pairs of leaf elements in the dataset can be merged to obtain a new dataset.

[0147] Then the center element and leaf element in the new dataset can be re-encoded, that is, the above process can be repeated. Figure 5 Steps two and three in the process continue until the change in the compression contribution is less than the threshold, at which point the iteration stops.

[0148] For example, such as Figure 11 As shown, after filtering out leaf element pairs that share the same central element, the central-leaf element pairs in the dataset can be merged. For example, if the filtered central-leaf element pairs are (a,b) and (a,c), and (b,b) and (b,c), then the leaf elements (a,b) and (a,c) in the dataset can be merged to obtain (a,{b,c}), and (b,b) and (b,c) can be merged to obtain (b,{b,c}). The positions of the merged patterns are then intersected. That is, the position of (a,{b,c}) is the intersection of the positions of (a,b) and (a,c), i.e., v1 and v5, and the position of (b,{b,c}) is the intersection of the positions of (b,b) and (b,c), i.e., v5. The non-intersecting parts of (b,b) and (b,c), such as the position value v4 of (b,b), can be retained.

[0149] For example, comparing the database encoding length after merging leaf elements can be done as follows: Figure 12 As shown. Leaf elements b and c can be merged, that is, elements with the same center element and leaf elements b and c are merged, and the merged patterns are encoded to obtain the corresponding encoding length.

[0150] Therefore, in this application's implementation, the strength of association between patterns can be represented by calculating the compression contribution, thereby enabling the discovery of patterns with stronger associations and improving the data mining effect. This can be understood as defining a star-shaped structure model and using Shannon entropy and conditional entropy principles to compress and encode the original data, allowing it to be represented by patterns that simultaneously satisfy high overall frequency and strong intrinsic association. Strong intrinsic association is reflected in the strong influence of the central element on the leaf elements. Compared to traditional frequent item pattern mining algorithms, the method provided in this application requires no extra parameters, avoids pattern explosion, has a high running speed, and can discover intrinsic associations beyond frequency. Compared to traditional compressed pattern mining algorithms, the method provided in this application is suitable for mining more complex data structures and can be applied to both graph data and transaction data. Furthermore, the resulting patterns are more universal; for graph data, it can classify attribute patterns with subtle differences in actual topological connection structures but the same properties into the same pattern. Moreover, the method provided in this application achieves good compression results while maintaining high running performance on complex, large datasets.

[0151] The foregoing has provided a detailed description of the data processing method provided in this application. The following section provides an exemplary description of the application scenarios and the effects achieved by the data processing method provided in this application.

[0152] Scenario 1: Equipment Alarm

[0153] The data processing method provided in this application can be deployed in a network maintenance center. When one or more devices in the network malfunction, an alarm storm is triggered. The network center receives a massive number of alarms and uses the method provided in this application to analyze historical data and derive correlation rules to correlate the alarms, identify important alarms, and filter or collapse non-important alarms.

[0154] By collecting alarms generated by various devices from historical data, and treating each type of alarm as a central element and alarms occurring within the same time period as leaf elements, strong central-leaf element pairs can be identified. When a device triggers an alarm again, the identified relationships can be used to predict upcoming or already detected alarms, allowing for timely responses to device alarms and improving device security.

[0155] For example, alarm correlation scenarios for a single device can be like this: Figure 13 As shown, multiple devices (such as...) can be used. Figure 13The alarm information of devices 1, 2, and 3 shown in the diagram is collected (i.e., the first dataset is obtained). Using the data processing method provided in this application, each alarm is treated as a central element, and alarms associated with the central element are treated as leaf elements. The correlation between alarms generated by multiple devices is mined, and alarms with strong correlations within the same device are identified and output (i.e., the final output second dataset). For example, when device 1 generates alarm POWER_ABNORMAL, based on the output correlation rules, it is determined that there is a high probability that alarms ETH_LOS and LTI will also be generated simultaneously, thus enabling timely alarm prediction.

[0156] For example, in scenarios involving alarm correlation between multiple devices, it can be done as follows: Figure 14 As shown, alarm information from multiple devices can be collected. Using the data processing method provided in this application, each alarm is treated as a central element, and alarms associated with the central element are treated as leaf elements. The correlation between alarms generated by different devices is then mined, revealing which alarms are likely to occur on other devices when certain alarms occur on one device. For example, when device 1 experiences a POWER_ABNORMAL alarm, based on the mined correlation rules, it can be predicted that device 2 may experience a MW_LOF alarm, and device 3 is also likely to experience a MW_LOF alarm.

[0157] Scenario 2: User Profile

[0158] The dataset processing method provided in this application can be applied to user profiling, and more accurate recommendations can be made to users based on these profiles. For example, the data processing method provided in this application can be deployed in systems such as social networks and e-commerce platforms, and can uncover relationships between user characteristics to achieve corresponding recommendation functions.

[0159] For example, user profiles can be created, including various user tags, such as the user's interests in music and movies. Multiple user profiles can be collected, and using the method provided in this application, each tag for each user can be used as a central element, and tags appearing simultaneously with the central element can be used as leaf elements. Strongly related element pairs can then be identified. This allows for the recommendation of more relevant tags to users during subsequent recommendations based on user profiles. Alternatively, when enriching user profiles, existing user tags can be referenced to obtain other tags with higher relevance.

[0160] For example, taking music recommendation as an example, an application scenario for recommending music to a specific user could be as follows: Figure 15As shown, the types of music played by multiple users can be collected (i.e., the first dataset is obtained). Then, using the method provided in this application, the correlation between various music types can be analyzed, and strongly related music types can be discovered. For example, a user who plays Rap may also play Rock and Rap, so Rock and Rap related music types can be recommended to the user.

[0161] For example, taking music recommendation as an example, application scenarios that recommend music to multiple users can be as follows: Figure 16 As shown, the music types played by multiple users can be collected (i.e., a first dataset is obtained), and the correlation between the music types of various users can be analyzed using the method provided in this application. For example, a user who plays rap might have friends who play rock and rap, thus recommending rock and rap-related music to that user's friends.

[0162] The data processing methods provided in this application can also be used to supplement the attributes of existing nodes. For example, for user profiles with few tags, information from a large number of existing user profiles can be used to analyze the correlation between tags, thereby mining tag pairs with strong correlations. These mined tag pairs can then be used to supplement user profiles with limited information, resulting in more comprehensive user profiles. As another example, in a music recommendation system, the data processing methods provided in this application can be used to mine the correlation between music types played by users, thereby predicting the music types played by users with limited information and achieving cold start in recommendation scenarios.

[0163] Furthermore, the data processing method provided in this application can also be applied to more scenarios, such as: most alarm / event correlation analysis, fault root cause location, etc.; in the field of CT intelligent operation and maintenance, such as fault root cause location, alarm correlation, etc.; in the field of IT data center intelligent operation and maintenance, such as fault root cause location, alarm handling, etc.; in the field of IT cloud service intelligent operation and maintenance, such as microservice application fault correlation analysis, fault root cause location, etc. This application is merely an illustrative example and will not elaborate on each one.

[0164] For example, the mining analysis of this application can be performed on several benchmark graph data mining datasets, such as DBLP (Digital Bibliography & Library Project), DBLP-Trend (Digital Bibliography Index and Library Project Trend Data), USFlight (US Flight Data), or Pokec (Slovakian Social Network), using the data processing methods provided in this application. See Table 4 for details.

[0165] Dataset DBLP DBLP-Trend USFlight Pokec Number of nodes 2723 2729 280 1632803 Total edges 3464 3464 4030 30622564 Number of central attributes 127 271 70 914 Category Quote Quote airport music

[0166] Table 4

[0167] Clearly, the data processing method provided in this application can extract a large amount of highly correlated data for different datasets.

[0168] The foregoing has provided a detailed description of the data processing method provided in this application. The apparatus provided in this application will now be described.

[0169] The foregoing has described the method flow provided in this application. The structure of the apparatus for performing the foregoing method flow is described below.

[0170] See Figure 17 This application provides a schematic diagram of the structure of a data processing device, which includes:

[0171] Module 1701 is used to obtain a first dataset, which includes multiple elements. Each of the multiple elements is used as a central element, and each central element is associated with at least one leaf element.

[0172] Processing module 1702 is used to obtain the encoding length of at least one central element and obtain the first encoded data of each central element. The encoding length of at least one central element is related to the number of times the at least one central element appears in the first dataset.

[0173] The processing module 1702 is further configured to obtain the encoding length of each leaf element associated with each center element based on the first encoding data, and obtain the second encoding data of each leaf element. The encoding length of each leaf element associated with each center element is related to the number of times each center element and each leaf element associated with each center element appear in the first dataset.

[0174] The processing module 1702 is also used to update the first dataset according to the first encoded data of each central element and the second encoded data of each leaf element to obtain the second dataset.

[0175] In one possible implementation, the processing module 1702 is specifically used to select at least two leaf elements that are associated with the same central element from the first dataset and merge them to obtain a second dataset, based on the first encoded data of each central element and the second encoded data of each leaf element.

[0176] In one possible implementation, the processing module 1702 is specifically configured to: use the first encoded data of each central element and the second encoded data of each leaf element to calculate the compression contribution rate of each set of leaf element pairs in the multiple sets of leaf element pairs, the compression contribution rate being used to represent the change in description length when each set of leaf element pairs is merged, wherein each set of leaf element pairs includes at least two leaf elements in the first dataset that have the same central element; and select at least one set of leaf element pairs from the multiple sets of leaf element pairs for merging based on the compression contribution rate of each set of leaf element pairs to obtain the second dataset.

[0177] In one possible implementation, the processing module 1702 is further configured to: if the compression contribution rate of multiple sets of leaf element pairs does not meet the preset conditions, then use the second dataset as the new first dataset, and re-encode the center element and leaf element in the new first dataset respectively; if the compression contribution rate of multiple sets of leaf element pairs meets the preset conditions, then stop updating the first dataset.

[0178] In one possible implementation, the encoding length of the first encoded data is negatively correlated with the number of times each central element appears in the first dataset.

[0179] In one possible implementation, the ratio between the first count and the second count is negatively correlated with the encoding length of the second encoded data for each leaf element. The first count is the number of times each leaf element in the first dataset appears simultaneously with its associated central element, and the second count is the number of times the central element associated with each leaf element in the first dataset appears alone.

[0180] In one possible implementation, the acquisition module 1701 is specifically used to: acquire input data, which includes multiple elements and the relationships between elements; construct a flipped database by taking each element as a center element and the elements associated with each center element as leaf elements, the flipped database including at least three columns, the at least three columns including the center element, the leaf elements and the columns formed by the positions of the center elements in the input data, and the first dataset including the flipped database.

[0181] In one possible implementation, the multiple elements in the first dataset include alarm information for each of the multiple devices, or the multiple elements in the first dataset include multiple alarm information for a single device, and the second dataset includes the correlation between the multiple alarms.

[0182] In one possible implementation, the acquisition module 1701 is further configured to acquire a first tag of the first user; and acquire a second tag associated with the first tag based on a second dataset.

[0183] In one possible implementation, the multiple elements in the first dataset include tags for multiple users; or, the multiple elements in the first dataset include multiple tags for a single user, and the second dataset includes the relationships between the multiple tags.

[0184] In one possible implementation, the acquisition module 1701 is further configured to acquire a first tag of the first user; and acquire a second tag associated with the first tag based on a second dataset.

[0185] Please see Figure 18 The following is a schematic diagram of another data processing device provided in this application.

[0186] The data processing device may include a processor 1801 and a memory 1802. The processor 1801 and the memory 1802 are interconnected via a circuit. The memory 1802 stores program instructions and data.

[0187] The aforementioned are stored in memory 1802 Figures 3-16 The steps in the code include the corresponding program instructions and data.

[0188] Processor 1801 is used to perform the aforementioned Figures 3-16 The method steps performed by the data processing apparatus shown in any of the embodiments.

[0189] Optionally, the data processing apparatus may also include a transceiver 1803 for receiving or sending data.

[0190] This application also provides a computer-readable storage medium storing a program for generating vehicle speed, which, when used on a computer, causes the computer to execute the aforementioned... Figures 3-16 The steps in the method described in the illustrated embodiment.

[0191] Alternatively, the aforementioned Figure 18 The data processing device shown is a chip.

[0192] This application embodiment also provides a data processing device, which may also be referred to as a digital processing chip or a chip. The chip includes a processing unit and a communication interface. The processing unit obtains program instructions through the communication interface, and the program instructions are executed by the processing unit. The processing unit is used to perform the aforementioned... Figures 3-16 The method steps performed by the data processing apparatus shown in any of the embodiments.

[0193] This application also provides a digital processing chip. This digital processing chip integrates circuitry for implementing the processor 1801 described above, or the functions of processor 1801, and one or more interfaces. When the digital processing chip integrates a memory, it can complete the method steps of any one or more of the foregoing embodiments. When the digital processing chip does not integrate a memory, it can be connected to an external memory via a communication interface. The digital processing chip implements the actions performed by the data processing device in the foregoing embodiments based on the program code stored in the external memory.

[0194] This application also provides a computer program product that, when run on a computer, causes the computer to perform the aforementioned actions. Figures 3-16 The steps performed by the data processing apparatus in the method described in the illustrated embodiment.

[0195] The data processing device provided in this application embodiment can be a chip, which includes a processing unit and a communication unit. The processing unit can be, for example, a processor, and the communication unit can be, for example, an input / output interface, pins, or circuits. The processing unit can execute computer execution instructions stored in a storage unit to cause the chip within the server to perform the aforementioned operations. Figures 3-16 The neural network training method described in the illustrated embodiment. Optionally, the storage unit is a storage unit within the chip, such as a register or cache. Alternatively, the storage unit can be a storage unit located outside the chip within the wireless access device, such as a read-only memory (ROM) or other types of static storage devices capable of storing static information and instructions, such as random access memory (RAM).

[0196] Specifically, the aforementioned processing unit or processor can be a central processing unit (CPU), a neural-network processing unit (NPU), a graphics processing unit (GPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0197] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0198] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0199] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0200] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0201] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in a sequence other than that illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0202] Finally, it should be noted that the above are only specific embodiments of this application, but the protection scope of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the protection scope of this application.

Claims

1. A data processing method, characterized in that, include: Obtain a first dataset, which includes multiple elements. Each of the multiple elements is used as a center element, and the elements associated with each center element are used as leaf elements. When an element is a center element and the center element has associated leaf elements, the center element is also used as a leaf element of the associated element. Obtain the encoding length of at least one central element, and obtain the first encoded data of each central element among the at least one central element. The encoding length of the at least one central element is related to the number of times the at least one central element appears in the first dataset. The encoding length of each leaf element associated with each center element is obtained based on the first encoded data, and the second encoded data of each leaf element is obtained. The encoding length of each leaf element associated with each center element is related to the number of times each center element and each leaf element associated with each center element appear simultaneously in the first dataset and the number of times each center element appears in the first dataset. The first dataset is updated based on the first encoded data of each central element and the second encoded data of each leaf element to obtain the second dataset; The first dataset includes alarm information or user profile tags, while the second dataset is used for device alarms or to determine user profiles, which are then used to make recommendations to users.

2. The method according to claim 1, characterized in that, The step of updating the first dataset according to the first encoded data of each center element and the second encoded data of each leaf element to obtain the second dataset includes: Based on the first encoded data of each central element and the second encoded data of each leaf element, at least two leaf elements that are associated with the same central element are selected from the first dataset and merged to obtain the second dataset.

3. The method according to claim 2, characterized in that, The step of selecting at least two leaf elements associated with the same central element from the first dataset and merging them to obtain the second dataset, based on the first encoded data of each central element and the second encoded data of each leaf element, includes: Using the first encoded data of each central element and the second encoded data of each leaf element, calculate the compression contribution rate of each leaf element pair in the multiple sets of leaf element pairs. The compression contribution rate is used to represent the change in the description length when each set of leaf element pairs is merged. Each set of leaf element pairs includes at least two leaf elements in the first dataset that have the same central element. Based on the compression contribution rate of each group of leaf element pairs, at least one group of leaf element pairs is selected from the multiple groups of leaf element pairs and merged to obtain the second dataset.

4. The method according to claim 3, characterized in that, The method further includes: If the compression contribution rate of the multiple sets of leaf element pairs does not meet the preset conditions, then the second dataset is used as the new first dataset, and the center element and leaf element in the new first dataset are re-encoded respectively. If the compression contribution rate of the multiple sets of leaf element pairs meets the preset condition, then the update of the first dataset is stopped.

5. The method according to any one of claims 1-4, characterized in that, The encoding length of the first encoded data is negatively correlated with the number of times each central element appears in the first dataset.

6. The method according to any one of claims 1-4, characterized in that, The ratio between the first count and the second count is negatively correlated with the encoding length of the second encoded data of each leaf element. The first count is the number of times each leaf element and its associated center element appear simultaneously in the first dataset, and the second count is the number of times the center element associated with each leaf element appears alone in the first dataset.

7. The method according to any one of claims 1-4, characterized in that, The process of obtaining the first dataset includes: Obtain input data, which includes various elements and the relationships between the elements; A flipped database is constructed by taking each of the multiple elements as a center element and the elements associated with each center element as leaf elements. The flipped database includes at least three columns, which include the center element, leaf elements, and a column formed by the position of the center element in the input data. The first dataset includes the flipped database.

8. The method according to any one of claims 1-4, characterized in that, The first dataset contains multiple elements including alarm information for each of the multiple devices, or the first dataset contains multiple alarm information for a single device, and the second dataset includes the correlation between the multiple alarms.

9. The method according to claim 8, characterized in that, The method further includes: Acquire the first alarm generated by the first device; Obtain the second alarm related to the first alarm based on the second dataset.

10. The method according to any one of claims 1-4, characterized in that, The first dataset contains multiple elements including tags for multiple users; or, the first dataset contains multiple elements including multiple tags for a single user, and the second dataset includes the relationships between multiple tags.

11. The method according to claim 10, characterized in that, The method further includes: Obtain the first user's first tag; Obtain the second label associated with the first label based on the second dataset.

12. A data processing apparatus, characterized in that, include: The acquisition module is used to acquire a first dataset, which includes multiple elements. Each of the multiple elements is used as a center element, and the elements associated with each center element are used as leaf elements. When an element is a center element and the center element has associated leaf elements, the center element is also used as a leaf element of the associated element. The processing module is used to obtain the encoding length of at least one central element and obtain the first encoded data of each central element among the at least one central element. The encoding length of the at least one central element is related to the number of times the at least one central element appears simultaneously in the first dataset and the number of times each central element appears in the first dataset. The processing module is further configured to obtain the encoding length of each leaf element associated with each center element based on the first encoded data, and obtain the second encoded data of each leaf element. The encoding length of each leaf element associated with each center element is related to the number of times each center element and each leaf element associated with each center element appear in the first dataset. The processing module is further configured to update the first dataset according to the first encoded data of each central element and the second encoded data of each leaf element to obtain the second dataset; The first dataset includes alarm information or user profile tags, while the second dataset is used for device alarms or to determine user profiles, which are then used to make recommendations to users.

13. The apparatus according to claim 12, characterized in that, The processing module is specifically used to select at least two leaf elements that are associated with the same central element from the first dataset and merge them to obtain the second dataset, based on the first encoded data of each central element and the second encoded data of each leaf element.

14. The apparatus according to claim 13, characterized in that, The processing module is specifically used for: Using the first encoded data of each central element and the second encoded data of each leaf element, calculate the compression contribution rate of each leaf element pair in the multiple sets of leaf element pairs. The compression contribution rate is used to represent the change in the description length when each set of leaf element pairs is merged. Each set of leaf element pairs includes at least two leaf elements in the first dataset that have the same central element. Based on the compression contribution rate of each group of leaf element pairs, at least one group of leaf element pairs is selected from the multiple groups of leaf element pairs and merged to obtain the second dataset.

15. The apparatus according to claim 14, characterized in that, The processing module is further configured to: If the compression contribution rate of the multiple sets of leaf element pairs does not meet the preset conditions, then the second dataset is used as the new first dataset, and the center element and leaf element in the new first dataset are re-encoded respectively. If the compression contribution rate of the multiple sets of leaf element pairs meets the preset condition, then the update of the first dataset is stopped.

16. The apparatus according to any one of claims 12-15, characterized in that, The encoding length of the first encoded data is negatively correlated with the number of times each central element appears in the first dataset.

17. The apparatus according to any one of claims 12-15, characterized in that, The ratio between the first count and the second count is negatively correlated with the encoding length of the second encoded data of each leaf element. The first count is the number of times each leaf element and its associated center element appear simultaneously in the first dataset, and the second count is the number of times the center element associated with each leaf element appears alone in the first dataset.

18. The apparatus according to any one of claims 12-15, characterized in that, The acquisition module is specifically used for: Obtain input data, which includes various elements and the relationships between the elements; A flipped database is constructed by taking each of the multiple elements as a center element and the elements associated with each center element as leaf elements. The flipped database includes at least three columns, which include the center element, leaf elements, and a column formed by the position of the center element in the input data. The first dataset includes the flipped database.

19. The apparatus according to any one of claims 12-15, characterized in that, The first dataset contains multiple elements including alarm information for each of the multiple devices, or the first dataset contains multiple alarm information for a single device, and the second dataset includes the correlation between the multiple alarms.

20. The apparatus according to claim 19, characterized in that, The acquisition module is also used for: Obtain the first user's first tag; Obtain the second label associated with the first label based on the second dataset.

21. The apparatus according to any one of claims 12-15, characterized in that, The first dataset contains multiple elements including tags for multiple users; or, the first dataset contains multiple elements including multiple tags for a single user, and the second dataset includes the relationships between multiple tags.

22. The apparatus according to claim 21, characterized in that, The acquisition module is also used for: Obtain the first user's first tag; Obtain the second label associated with the first label based on the second dataset.

23. A data processing apparatus, characterized in that, The method includes one or more processors coupled to a memory storing a program that, when executed by the one or more processors, implements the steps of any one of claims 1 to 11.

24. A computer-readable storage medium, characterized in that, The program, when executed by the processing unit, performs the method as described in any one of claims 1 to 11.

25. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method as described in any one of claims 1 to 11.

Images