Data analysis methods, devices, and computer storage media for skewed partitioning

Abstract

The present application discloses a data analysis method, device and computer storage medium for tilted partitions, and the detection method includes: in response to an analysis request, obtaining a tilted partition; based on the frequency of occurrence of keywords in each data record in the tilted partition, obtaining the record keywords and their frequency of occurrence in the mapping table; when a new keyword appears in the data record of the tilted partition, using the top heap sorting result corresponding to the mapping table to delete the keyword with the least frequency of occurrence, so as to write the new keyword into the mapping table; after the data record analysis of the tilted partition is completed, based on the final keyword of the mapping table and its frequency of occurrence, obtaining the data analysis result of the tilted partition. In the above manner, the present application can quickly record and count each keyword in the tilted partition by performing data analysis on the tilted partition, and achieves the effect of analyzing the specific data distribution of each keyword in the tilted partition.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a data analysis method and device for tilted partitions and a computer storage medium. Background Art

[0002] In distributed computing, skewed partitioning means that during the parallel processing of tasks, the amount of data processed by some partitions or nodes is significantly higher than that of other partitions or nodes. For example, if a Spark (Apache Spark, a distributed computing framework) job involves Shuffle (the process of data repartitioning), partitioning will be performed based on the hash value of the Shuffle Key. If a partitioned data set contains a large number of duplicate Shuffle Keys, the amount of data in the partition will be significantly larger than that in other partitions, that is, the partition is skewed.

[0003] The detection of tilted partitions is usually based on whether the time taken by tasks or jobs in each partition exceeds the expected value to determine whether it is a tilted partition.

[0004] In the prior art, tilted partitions can usually only be judged based on the running time of the partition tasks, and no further analysis is performed on the tilted partition data during the detection process, so the data distribution of the tilted partition cannot be obtained. Summary of the invention

[0005] The main technical problem solved by the present application is how to quickly analyze the data partitioning status of tilted partitions. To this end, the present application provides a data analysis method, device and computer storage medium for tilted partitions.

[0006] In order to solve the above technical problems, a technical solution adopted by the present application is to provide a data analysis method for tilted partitions, the method comprising:

[0007] In response to the analysis request, obtaining the tilted partition;

[0008] Based on the occurrence frequency of the keyword in each data record in the tilted partition, obtaining the record keyword and its occurrence frequency in the mapping table;

[0009] When a new keyword appears in the data record of the tilted partition, the keyword with the least frequency of occurrence is deleted using the top heap sorting result corresponding to the mapping table, so as to write the new keyword into the mapping table;

[0010] After the data record analysis of the inclined partition is completed, the data analysis result of the inclined partition is obtained based on the final keyword of the mapping table and the frequency of occurrence thereof.

[0011] Among them, the data analysis method also includes: when a keyword appears in the data record of the tilted partition, determining whether the keyword in each data record in the tilted partition appears in the mapping table record; if so, accumulating the frequency of occurrence of the keyword in the mapping table record; if not, adding the keyword to the mapping table record.

[0012] Among them, when a new keyword appears in the data record of the inclined partition, the keyword with the least frequency of occurrence is deleted using the top heap sorting result corresponding to the mapping table to write the new keyword into the mapping table, including: when a new keyword appears in the data record of the inclined partition, the real-time record length of the mapping table is obtained; whether the real-time record length exceeds a preset length threshold; if so, the keyword with the least frequency of occurrence is deleted using the top heap sorting result corresponding to the mapping table to write the new keyword into the mapping table; if not, the new keyword is written into the mapping table.

[0013] Among them, the method of deleting the keyword with the least frequency of occurrence by using the top heap sorting result corresponding to the mapping table includes: when the type of the top heap is a small top heap, obtaining the heap sorting result of the small top heap corresponding to the mapping table, and deleting the first keyword in the heap sorting result in a front-to-back order.

[0014] Among them, the final keywords and their occurrence frequencies based on the mapping table are used to obtain the data analysis results of the inclined partition, including: obtaining the number of occurrences of the final keywords recorded in the mapping table in the inclined partition and the length of the final keywords; obtaining the total number of keyword occurrences of all final keywords; based on the number of occurrences of each final keyword and the total number of keyword occurrences, generating the number of occurrences ratio of each final keyword; and using the number of occurrences ratio and the length of each final keyword to generate the data analysis results.

[0015] Among them, the obtaining of tilted partitions includes: obtaining a data set to be analyzed; obtaining a key-value pair data set output after the mapping table executes the data set to be analyzed; partitioning and sorting the key-value pair data set to obtain a sorted partitioned data set; counting the data size of the sorted partitioned data set to obtain statistical information corresponding to each partition; and using the statistical information corresponding to each partition to analyze the tilted partitions.

[0016] Among them, using the statistical information corresponding to each partition to analyze the tilted partition includes: judging whether the statistical information meets the preset tilt judgment condition; if so, taking the partition that meets the tilt judgment condition as the tilted partition; if not, determining that the partition corresponding to the statistical information does not belong to the tilted partition.

[0017] Among them, the judging whether the statistical information satisfies the preset tilt judgment condition includes: obtaining the historical partition size information of the historical cache partition and the current partition size information of the current partition based on the statistical information; obtaining the partition average threshold based on the historical partition size information; judging whether the current partition size information satisfies the partition average threshold and the preset data size threshold; if so, the current partition that satisfies the partition average threshold and the preset data size threshold is regarded as a tilted partition; if not, determining that the current partition does not belong to a tilted partition; wherein the tilt judgment condition includes the partition average threshold and the preset data size threshold.

[0018] To solve the above-mentioned technical problems, another technical solution adopted in the present application is: to provide a data analysis device for tilted partitions, the data analysis device for tilted partitions comprising a processor and a memory, the memory being coupled to the processor, the memory storing program data, and the processor being used to execute the program data to implement the data analysis method for tilted partitions as mentioned above.

[0019] In order to solve the above technical problems, another technical solution adopted by the present application is: to provide a computer-readable storage medium, which stores program data, and when the program data is executed, it is used to implement the above-mentioned data analysis method for tilted partitions.

[0020] The beneficial effects of the present application are as follows: different from the prior art, the data analysis method for inclined partitions provided by the present application is applied to a data analysis device for inclined partitions, and the data analysis device for inclined partitions obtains the inclined partitions in response to an analysis request; based on the occurrence frequency of keywords in each data record in the inclined partition, the record keywords and their occurrence frequency in the mapping table are obtained; when new keywords appear in the data records of the inclined partition, the keywords with the least occurrence frequency are deleted using the top heap sorting result corresponding to the mapping table, so as to write the new keywords into the mapping table; after the data record analysis of the inclined partition is completed, the data analysis results of the inclined partition are obtained based on the final keywords in the mapping table and their occurrence frequency. In the above manner, compared with the conventional data analysis method for inclined partitions, the data analysis method for inclined partitions provided by the present application can quickly record and count each keyword in the inclined partition, and realize the analysis of the specific data distribution of each keyword in the inclined partition. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. Among them:

[0022] Figure 1 It is a flow chart of an embodiment of a data analysis method for inclined partitions provided by the present application;

[0023] Figure 2 It is a flowchart of another embodiment of the data analysis method for inclined partitions provided by the present application;

[0024] Figure 3 This is a flow chart of a first embodiment of recording the frequency of occurrence of keywords in tilted partitions provided by the present application;

[0025] Figure 4 It is a flow chart of the frequency of occurrence of keywords in the statistical tilt partition provided by this application;

[0026] Figure 5 This is a graph showing the relationship between Map length and execution time (ms) when the number of keywords provided by this application is not a fixed value;

[0027] Figure 6 This is a graph showing the relationship between Map length and execution time (ms) when the number of keywords provided by this application is a fixed value;

[0028] Figure 7 This is a flow chart of a second embodiment of recording the frequency of occurrence of keywords in tilted partitions provided by the present application;

[0029] Figure 8 It is a structural schematic diagram of an embodiment of a data analysis device for inclined partitions provided by the present application;

[0030] Fig. 9 It is a structural schematic diagram of another embodiment of the data analysis device for inclined partitions provided by the present application;

[0031] Fig.10 It is a structural schematic diagram of an embodiment of a computer-readable storage medium provided by the present application. DETAILED DESCRIPTION

[0032] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0033] The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the objects associated before and after are in an "or" relationship. In addition, "many" in this article means two or more than two. In addition, the term "at least one" in this article means any combination of at least two of any one or more of a plurality of, for example, including at least one of A, B, and C, can mean including any one or more elements selected from the set consisting of A, B, and C.

[0034] The data analysis method for tilted partitions provided in the present application is mainly applied to a data analysis device for tilted partitions, wherein the data analysis device for tilted partitions of the present application can be a server, or a system composed of a server and a terminal device cooperating with each other. Accordingly, the various parts of the data analysis device for tilted partitions, such as various units, subunits, modules, and submodules, can all be set in the server, or can be set in the server and the terminal device respectively.

[0035] Furthermore, the above-mentioned server can be hardware or software. When the server is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or it can be implemented as a single server. When the server is software, it can be implemented as multiple software or software modules, such as software or software modules used to provide distributed servers, or it can be implemented as a single software or software module, which is not specifically limited here. In some possible implementations, the data analysis method of the tilted partition of the embodiment of the present application can be implemented by a processor calling a computer-readable instruction stored in a memory.

[0036] In the prior art, tilted partitions can usually only be judged based on the running time of the partition tasks, and no further analysis is performed on the tilted partition data during the detection process, so the data distribution of the tilted partition cannot be obtained.

[0037] Therefore, the present application provides a data analysis method for tilted partitions, which achieves efficient data analysis by analyzing the data distribution of tilted partitions. The specific implementation method is described as follows.

[0038] See also Figure 1 , Figure 1 It is a flow chart of an embodiment of a data analysis method for tilted partitions provided in the present application.

[0039] Step 11: In response to the analysis request, obtain the tilted partition.

[0040] Specifically, the analysis request may be a request issued by a terminal or a user, which is determined based on an actual scenario.

[0041] Specifically, the skewed partition refers to the imbalanced data distribution caused by the fact that the amount of data in some partitions is significantly larger than that in other partitions due to the natural distribution of data, business logic or other factors during the data partitioning process.

[0042] In some embodiments, obtaining the tilted partition includes: obtaining a data set to be analyzed; obtaining a key-value pair data set output after the mapping table executes the data set to be analyzed; partitioning and sorting the key-value pair data set to obtain a sorted partitioned data set; counting the data size of the sorted partitioned data set to obtain statistical information corresponding to each partition; and analyzing the tilted partition using the statistical information corresponding to each partition.

[0043] The data set to be analyzed refers to a set of data tasks that Shuffle Map needs to execute; the key-value pair data set refers to the key-value pairs parsed after Shuffle Map executes the data tasks.

[0044] Specifically, the Map Stage reads the data set to be analyzed from the shard to which it is assigned, and parses each key-value of the data set to be analyzed in the shard. Map is a mapping table.

[0045] Specifically, the key-value pair data set is partitioned by using the partition function of the Map Stage to determine the partition data set, and the data in the partition data set is sorted by using a quick sort algorithm to obtain a sorted partition data set.

[0046] Specifically, after the Map Stage is completed, the data quantity and data size of each partition generated are counted to generate statistical information corresponding to each partition.

[0047] See the procedure for obtaining statistics and tilted partitions. Figure 2 , Figure 2 It is a flowchart of another embodiment of the data analysis method for tilted partitions provided in the present application.

[0048] Specifically, before generating statistics in the Shuffle Map stage, the type and column order information of the join key are written into the Shuffle Dependecy (which describes how data is transferred and redistributed between stages). Furthermore, after the MapStage ends, the statistics output by the cached Map shards are managed by the MapOutputTrackerMaster component of the Driver (driver / driver process) through the Get Statistics function.

[0049] Among them, Shuffle Dependency is a class in Spark that represents the dependency of the Shuffle process. It is mainly used to describe the data dependencies between different stages in a Spark job, especially the dependencies related to the Shuffle operation, which helps Spark to correctly schedule tasks to ensure that data is correctly transmitted and processed between stages.

[0050] Specifically, the Driver is the main component of the Spark application and is responsible for the execution and coordination of the entire application. Its main function is to convert the program into jobs or tasks, schedule tasks between Executors in Spark, and track the execution of Executors, which helps to improve the accuracy of subsequent tilt partition judgment.

[0051] Driver includes MapOutputTrackerMaster (MapOutputTrackerMasterEndpoint, task output tracker) component and DAGScheduler (Directed Acyclic Graph Scheduler, task scheduler) component; among them, MapOutputTrackerMaster component is used to optimize the local scheduling of program tasks based on statistical information; DAGScheduler component is Spark's task scheduler, which is responsible for detecting tasks and adjusting the execution order and dependencies of tasks as needed.

[0052] In some embodiments, the use of statistical information corresponding to each partition to analyze the tilted partition includes: determining whether the statistical information meets a preset tilt judgment condition; if so, taking the partition that meets the tilt judgment condition as the tilted partition; if not, determining that the partition corresponding to the statistical information does not belong to a tilted partition.

[0053] The preset tilt judgment condition is determined based on the actual needs of the user, and the user can flexibly select the judgment condition according to the needs.

[0054] Specifically, the tilt judgment condition may be a partition average threshold and a preset data size threshold.

[0055] For example, the partition average threshold may be a fixed multiple (such as four times, five times, etc.) of the average value of the historical partition data size, and the preset data size threshold may be the data volume size (such as 256).

[0056] In some embodiments, the determination of whether the statistical information satisfies a preset tilt determination condition includes: obtaining historical partition size information of a historical cache partition and current partition size information of a current partition based on the statistical information; obtaining a partition average threshold based on the historical partition size information; determining whether the current partition size information satisfies the partition average threshold and a preset data size threshold; if so, treating the current partition that satisfies the partition average threshold and the preset data size threshold as a tilted partition; if not, determining that the current partition does not belong to a tilted partition.

[0057] The historical partition size information of the historical cache partition refers to the size of the data volume of the historical cache partition in the statistical information.

[0058] Specifically, the partition average threshold refers to a fixed multiple of the average value of the historical partition data size in the statistical information.

[0059] For details on the process of determining tilted partitions, see Figure 2 .

[0060] Specifically, in the Shuffle Read stage, it is necessary to determine whether the current partition is a skewed partition. The determination method is to obtain and compare the historical partition size information of each historical cache partition from the Driver through the MapOutputTrackerWorker component in the Executor. If the current partition size information exceeds 4 times the average value of the historical partition data size and the data volume of the current partition size information is greater than the preset data size threshold, the current partition is considered to be a skewed partition.

[0061] Among them, Executor is responsible for executing specific tasks in Spark jobs: judging tilted partitions and counting keywords in tilted partitions, etc.; it is also responsible for communicating with the Driver node: receiving tasks and sending information such as running status, and is also responsible for detecting task execution status: including task execution progress, resource usage, etc.

[0062] In this embodiment, by judging whether the current partition size information meets the partition average threshold and the preset data size threshold, the judgment of the tilted partition can be realized by combining the two judgment conditions of the partition average threshold and the preset data size threshold, thereby improving the accuracy of the tilted partition judgment.

[0063] Step 12: Based on the occurrence frequency of the keyword in each data record in the tilted partition, obtain the record keyword and its occurrence frequency in the mapping table.

[0064] For details on the process of counting the frequency of occurrence of keywords in each data record in the tilted partition, please refer to Figure 2 .

[0065] Specifically, after the process of determining the skewed partition is completed, the skewed partition determination result is sent to the ShuffleRowRDD (Shuffle Row Record Data Set, elastic distributed data repartitioning component), and the statistics of keywords are implemented through the ShuffleRowRDD component.

[0066] Specifically, the keywords in each data record in the skewed partition are RDD (Resilient Distributed Dataset) data generated after the Shuffle operation. When counting the frequency of keyword occurrence, the ShuffleRowRDD component is used to count the frequency of keyword occurrence. The ShuffleRowRDD component can be used to count keywords in combination with Map and top heap operations.

[0067] For the specific statistical process of the frequency of occurrence of keywords in each data record in the tilted partition, refer to Figure 3 and Figure 4 , Figure 3 This is a flow chart of a first embodiment of recording the frequency of occurrence of keywords in tilted partitions provided by the present application; Figure 4 It is a flow chart of the frequency of occurrence of keywords in the statistical tilt partition provided by this application.

[0068] Step 31: When a keyword appears in the data record of the inclined partition, it is determined whether the keyword in each data record in the inclined partition appears in the mapping table record.

[0069] Specifically, Map is used to store the occurrence of join keys and the occurrence frequency corresponding to the join keys.

[0070] Specifically, each record in the Map record is traversed to find out whether each keyword in the tilted partition appears in the Map record. If so, proceed to step 32; if not, proceed to step 33.

[0071] Step 32: Accumulate the occurrence frequency of the keyword in the mapping table record.

[0072] Specifically, if a keyword in each data record in the tilted partition appears in a Map record, it is determined that the keyword is not a new keyword, and the occurrence frequency of the keyword can be directly added by one in the Map to record the occurrence frequency of the keyword.

[0073] Step 33: Add the keyword to the mapping table record.

[0074] Specifically, if the keyword in each data record in the tilted partition does not appear in the Map record, the keyword is determined to be a new keyword, and the new keyword can be added through the Map, and the occurrence frequency of the new keyword is recorded as once.

[0075] In this embodiment, by updating the recorded keywords and their occurrence frequencies in the Map based on the occurrence frequencies of keywords in each data record in the tilted partition, it is possible to record and count keywords, which facilitates subsequent analysis of data distribution in the tilted partition.

[0076] Step 13: When a new keyword appears in the data record of the tilted partition, the keyword with the least frequency of occurrence is deleted using the top heap sorting result corresponding to the mapping table, so as to write the new keyword into the mapping table.

[0077] Specifically, the top heap is used to quickly record keywords and their occurrence frequencies, and the top heap can be a small top heap, wherein the small top heap refers to a data structure that sorts data from small to large, and the top element of the heap is the smallest element in the data.

[0078] For details on the relationship between Map length and the execution time of tilted partitions, see Figure 5 and Figure 6 , Figure 5 This is a graph showing the relationship between Map length and execution time (ms) when the number of keywords provided by this application is not a fixed value; Figure 6 This is a graph showing the relationship between Map length and execution time (ms, millisecond) when the number of keywords provided in this application is a fixed value.

[0079] Specifically, the number of keywords in the tilted partition is obtained; if the number of keywords belongs to the first interval, the longer the Map length is, the execution time of executing the tilted partition gradually increases; if the number of keywords belongs to the second interval, the longer the Map length is, the execution time of executing the tilted partition gradually decreases; if the number of keywords is a constant, the closer the Map length is to the number of keywords, the execution time of executing the tilted partition gradually decreases; if the Map length is a constant, the larger the number of keywords is, the execution time of executing the tilted partition gradually increases.

[0080] Among them, the number of keywords (keys) determines the frequency of Map insertion operations and the frequency of heap sorting; the length of the key will affect the memory space. For example, if the key length is too long, it may cause excessive memory space occupation; the Map length determines the frequency of Map key removal operations and affects the memory size data occupied by the Map and the top heap.

[0081] Through the relationship between Map length and execution time described above, we can understand that in the process of counting keywords in tilted partitions, the Map configuration items can dynamically configure the Map length in real time based on the number of keywords, so as to select the optimal Map configuration items corresponding to different numbers of keywords under different numbers of keywords, thereby improving the efficiency of counting keywords in tilted partitions.

[0082] For the detailed process of configuring the top heap record keywords and their occurrence frequency, refer to Figure 7 , Figure 7 It is a flowchart of the second embodiment of recording the frequency of occurrence of keywords in tilted partitions provided by the present application.

[0083] Step 71: When a new keyword appears in the data record of the tilted partition, the real-time record length of the mapping table is obtained.

[0084] Specifically, the length of a Map is the number of key-value pairs that can be stored in the Map.

[0085] Step 72: Determine whether the real-time record length exceeds a preset length threshold.

[0086] Specifically, the preset length threshold of the Map can be determined based on actual user needs.

[0087] In this embodiment, by judging whether the Map length exceeds the Map length configuration item, it is not necessary to count all the data records in the tilted partition at once, and a certain number of data records can be selected for keyword recording and statistics, which helps to improve the efficiency of keyword recording and statistics. If yes, go to step 73; if no, go to step 74.

[0088] Step 73: Utilize the top heap sorting result corresponding to the mapping table to delete the keyword with the least occurrence frequency, so as to write the new keyword into the mapping table.

[0089] Specifically, the heap sort result is the result after sorting the keywords based on the top heap. The basis of the heap sort is determined by comparing the frequency of occurrence of two keywords in the Map record. If the top heap is a small top heap, the heap sort result is the result of ascending order.

[0090] Specifically, if the length of the Map exceeds the preset length threshold of the Map, it means that the length of the Map has reached the maximum. At this time, the keyword with the least frequency in the Map record is deleted to update the Map, and the updated keyword and its frequency of occurrence are recorded.

[0091] Specifically, the keyword with the least frequency of occurrence can be obtained by updating the top of the mini-heap.

[0092] Step 74: Write the new keyword into the mapping table.

[0093] Specifically, if the length of the Map does not exceed the Map length configuration item, it means that the length of the Map has not reached the maximum value, and the new keyword can be directly written into the Map.

[0094] In the embodiment of the present application, by setting the configuration items of the Map and the configuration items of the top heap, the user can flexibly select the optimal configuration phase to record and count keywords, thereby improving the accuracy of keyword recording and statistics. In combination with the configuration items of the top heap, it is possible to selectively record each record data in the tilted partition without having to record all the data at once, thereby saving recording time. The keyword with the least frequency of occurrence can be directly located through the top of the small top heap, thereby achieving rapid positioning of tilted keywords and improving the efficiency of recording and positioning keyword records.

[0095] Step 14: After the data record analysis of the inclined partition is completed, the data analysis result of the inclined partition is obtained based on the final keyword and the frequency of occurrence of the keyword in the mapping table.

[0096] Specifically, the data analysis results include but are not limited to the n most frequently occurring keywords and their proportion in the total number of keywords; wherein n is the number of the most frequently occurring keywords determined based on actual user needs.

[0097] In some embodiments, the final keyword and its occurrence frequency based on the Map are used to obtain the data analysis results of the tilted partition, including: obtaining the number of occurrences of the final keyword recorded in the Map in the tilted partition and the length of the final keyword; obtaining the total number of keyword occurrences of all final keywords; generating the number of occurrences ratio of each final keyword based on the number of occurrences of each final keyword and the total number of keyword occurrences; and generating the data analysis results using the number of occurrences ratio and the length of each final keyword.

[0098] Specifically, the sorting result of Map includes the number of all keywords in the tilted partition, the key length data, and the sorted keywords sorted by the number of keywords.

[0099] Specifically, based on the number of keywords and the length of the keywords, the specific quantity information and length information of each keyword can be understood to generate data analysis results including the number of keywords with the highest frequency of occurrence and the proportion of each keyword with the highest frequency of occurrence in the total number of keywords.

[0100] In the embodiment of the present application, by generating the proportion of each keyword in the total number of keywords based on the number of each keyword and the total number of keywords, it is possible to quickly analyze the specific data distribution of each keyword in the tilted partition.

[0101] In this embodiment, after obtaining the data analysis results of the tilted partition based on the final keywords and their occurrence frequencies of the Map, the method also includes: after the Executor executes the keyword statistical process, the statistical data analysis results will be sent to the Driver, and the DAGScheduler will write them into the Spark event log, so as to display the data analysis results of the tilted partition in the event log through the Spark UI or a custom diagnostic tool.

[0102] In this embodiment, the data analysis results of the skewed partitions in the event log are displayed through the Spark UI (User Interface) or a customized diagnostic tool, and the most frequently appearing keywords and their proportion data in the skewed partitions can be intuitively displayed to the user through the data analysis results, so that the user can directly analyze the root cause of the skewed partition, thereby effectively avoiding the occurrence of skewed partitions during program operation.

[0103] Different from the prior art, the data analysis method for inclined partitions provided by the present invention is applied to a data analysis device for inclined partitions. The data analysis device for inclined partitions obtains the inclined partitions in response to the analysis request; based on the occurrence frequency of keywords in each data record in the inclined partition, the record keywords and their occurrence frequency in the Map are obtained; when new keywords appear in the data records of the inclined partitions, the keywords with the least occurrence frequency are deleted using the top heap sorting result corresponding to the Map, so as to write the new keywords into the Map; after the data record analysis of the inclined partition is completed, the data analysis results of the inclined partition are obtained based on the final keywords of the Map and their occurrence frequency. Through the above manner, compared with the conventional data analysis method for inclined partitions, the data analysis method for inclined partitions provided by the present application can quickly record and count each keyword in the inclined partition, and realize the analysis of the specific data distribution of each keyword in the inclined partition.

[0104] The method of the above embodiment can be implemented by using a data analysis device for tilted partitions. Figure 8 Describe, Figure 8 It is a structural diagram of an embodiment of a data analysis device for tilted partitions provided in the present application.

[0105] like Figure 8As shown, the tilted partitioned data analysis device 80 of the embodiment of the present application includes an acquisition module 81, an update module 82, a configuration module 83 and an analysis module 84.

[0106] The acquisition module 81 is used to acquire the tilted partition in response to the analysis request.

[0107] The updating module 82 is used to obtain the record keywords and their occurrence frequencies in the mapping table based on the occurrence frequencies of the keywords in each data record in the tilted partition.

[0108] Configuration module 83 is used to delete the keyword with the least frequency of occurrence by using the top heap sorting result corresponding to the mapping table when a new keyword appears in the data record of the tilted partition, so as to write the new keyword into the mapping table.

[0109] The analysis module 84 is used to obtain the data analysis result of the inclined partition based on the final keyword and the occurrence frequency of the keyword in the mapping table after the data record analysis of the inclined partition is completed.

[0110] The method of the above embodiment can be implemented by using a data analysis device for inclined partitions. Fig. 9 , Fig. 9 It is a structural schematic diagram of another embodiment of the data analysis device for tilted partitions provided in the present application. The data analysis device 90 for tilted partitions includes a memory 91 and a processor 92. The memory 91 is used to store program data, and the processor 92 is used to execute program data to implement the data analysis method for tilted partitions in the above-mentioned embodiment.

[0111] The method of the above embodiment can be implemented using a computer-readable storage medium. Fig.10 , Fig.10 It is a structural diagram of an embodiment of a computer-readable storage medium provided in the present application. The computer-readable storage medium 100 stores program data 101. When the program data 101 is executed by a processor, it is used to implement the data analysis method of the tilted partition of the above embodiment.

[0112] When the embodiments of the present application are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

[0113] The above description is only an implementation method of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly used in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A data analysis method for tilted partitions, characterized in that: The data analysis method of the tilted partition includes: In response to the analysis request, obtaining the tilted partition; Based on the occurrence frequency of the keyword in each data record in the tilted partition, obtaining the record keyword and its occurrence frequency in the mapping table; When a new keyword appears in the data record of the tilted partition, the keyword with the least frequency of occurrence is deleted using the top heap sorting result corresponding to the mapping table, so as to write the new keyword into the mapping table; After the data record analysis of the inclined partition is completed, based on the final keyword of the mapping table and its occurrence frequency, the data analysis result of the inclined partition is obtained; The obtaining of the tilted partition includes: Get the data set to be analyzed; Obtaining a key-value pair data set outputted after the mapping table executes the data set to be analyzed; Partitioning and sorting the key value pair data set to obtain a sorted partitioned data set; Counting the data size of the sorted partition data set to obtain statistical information corresponding to each partition; Analyze the tilted partitions using the statistical information corresponding to each partition; The analyzing the tilted partition by using the statistical information corresponding to each partition includes: Determining whether the statistical information meets a preset tilt determination condition; If yes, the partition satisfying the tilt judgment condition is used as the tilt partition; If not, determining that the partition corresponding to the statistical information does not belong to a tilted partition; The determining whether the statistical information satisfies a preset tilt determination condition includes: Based on the statistical information, obtain historical partition size information of the historical cache partition and current partition size information of the current partition; Obtaining a partition average threshold based on the historical partition size information; Determine whether the current partition size information meets the partition average threshold and the preset data size threshold; If yes, the current partition that meets the partition average threshold and the preset data size threshold is used as a tilted partition; If not, it is determined that the current partition does not belong to a tilted partition.

2. The data analysis method for inclined partitions according to claim 1, characterized in that: The data analysis method further includes: When a keyword appears in the data record of the tilted partition, determining whether the keyword in each data record in the tilted partition appears in the mapping table record; If yes, accumulating the occurrence frequency of the keyword in the mapping table record; If not, the keyword is added to the mapping table record.

3. The data analysis method for inclined partitions according to claim 1, characterized in that: When a new keyword appears in the data record of the inclined partition, the keyword with the least frequency of occurrence is deleted using the top heap sorting result corresponding to the mapping table to write the new keyword into the mapping table, including: When a new keyword appears in the data record of the tilted partition, obtaining the real-time record length of the mapping table; Determining whether the real-time record length exceeds a preset length threshold; If yes, deleting the keyword with the least frequency of occurrence using the top heap sorting result corresponding to the mapping table, so as to write the new keyword into the mapping table; If not, write the new keyword into the mapping table.

4. The data analysis method for inclined partitions according to claim 3 is characterized in that: The deleting the keyword with the least frequency of occurrence using the top heap sorting result corresponding to the mapping table includes: When the type of the top heap is a small top heap, a heap sorting result of the small top heap corresponding to the mapping table is obtained, and the first keyword is deleted from the heap sorting result in a front-to-back order.

5. The data analysis method for inclined partitions according to claim 1, characterized in that: The obtaining of the data analysis result of the tilted partition based on the final keyword and the frequency of occurrence of the final keyword in the mapping table includes: Obtaining the number of occurrences of the final keyword and the length of the final keyword recorded in the mapping table in the tilted partition; Get the total number of keyword occurrences of all final keywords; Based on the number of occurrences of each final keyword and the total number of occurrences of the keyword, generating a ratio of the number of occurrences of each final keyword; The data analysis result is generated using the occurrence ratio and the length of each final keyword.

6. A data analysis device for tilted partitions, characterized in that: The data analysis device for tilted partitions includes a memory and a processor coupled to the memory; The memory is used to store program data, and the processor is used to execute the program data to implement the data analysis method for inclined partitions as described in any one of claims 1 to 5.

7. A computer storage medium, characterized in that: The computer storage medium is used to store program data, and when the program data is executed by a computer, it is used to implement the data analysis method for inclined partitions as described in any one of claims 1 to 5.

Images