A data skew point searching method and device, computer equipment and storage medium

Abstract

The application discloses a data skew point searching method and device, computer equipment and a storage medium, and belongs to the technical field of big data and the field of insurance and finance. The application determines a target object and a skew log field of data skew by means of an HQL execution log, obtains the association relation of the target object, marks the association relation to obtain an association identifier, determines a first data flow conversion relation corresponding to the skew log field, obtains a stage plan of the HQL execution log, determines a second data flow conversion relation corresponding to the skew log field, compares the first data flow conversion relation with the second data flow conversion relation, determines a flow conversion relation difference, determines the position of the flow conversion relation difference based on the association identifier, and obtains the position of the data skew point. In addition, the application also relates to the blockchain technology, and the HQL execution log can be stored in the blockchain. The scheme of the application can quickly locate the data skew point in the database, reduces the manpower investment, and improves the accuracy of data skew point searching.

Description

Technical Field

[0001] This application belongs to the fields of big data technology and insurance and finance, and specifically relates to a method, device, computer equipment and storage medium for finding data tilt points. Background Technology

[0002] For clustered systems, caching typically employs distributed caching, where different nodes are responsible for a specific range of cached data. In clustered systems, insufficient distribution of cached data often leads to a large amount of cached data being concentrated on one or a few service nodes, a phenomenon known as data skew. Generally, data skew is caused by poor load balancing; some nodes have more data than others. During computation, some nodes finish processing early, while others remain unprocessed, causing the entire processing flow to be delayed. This is the most direct manifestation of data skew.

[0003] If the data volume in each partition is relatively evenly distributed during computation, each executor can process the data in its assigned partition within a similar timeframe. However, in reality, under certain circumstances, the data in each partition can become extremely uneven during a particular computation stage, especially after a distributed partition shuffle process. This situation is commonly known as data skew. The direct consequence of data skew is that partitions with large amounts of data take extremely long to process. While other partitions have already completed their computations, some partitions with large amounts of data may still be unable to finish, resulting in low parallelism for the entire system and prolonged overall computation time. In some particularly severe cases, individual partitions may become so large that they run out of memory during computation, causing the executor executing the computation task to be terminated by the operating system due to memory overflow, and the overall computation task may never be completed.

[0004] In the big data industry, such as the insurance and finance industry, data skew is a common problem in some insurance and financial management systems. However, when dealing with data skew, manual processing is usually used. But because HQL logs are generally long and have high logical complexity, manual processing of data skew requires a lot of effort, and manual processing is prone to errors and has low accuracy. Summary of the Invention

[0005] The purpose of this application is to provide a method, apparatus, computer device, and storage medium for finding data skew points, so as to solve the technical problems that existing manual processing of data skew problems requires a lot of effort, is prone to errors, and has low accuracy.

[0006] To address the aforementioned technical problems, this application provides a method for finding data skew points, employing the following technical solution:

[0007] A method for finding data skew points includes:

[0008] Obtain the HQL execution log, and identify the target object where data skew occurs and the skew log field in the HQL execution log;

[0009] Obtain the association relationships of the target object, and mark the association relationships in the HQL execution log to obtain the association identifier;

[0010] Parse the HQL execution plan and determine the first data flow relationship corresponding to the skew log field based on the parsing results;

[0011] Obtain the phase plan of the HQL execution log, and determine the second data flow relationship corresponding to the skew log field from the phase plan;

[0012] Compare the first data flow relationship with the second data flow relationship to determine the differences in the flow relationship;

[0013] Based on the association identifier, the location of the difference in the flow relationship is determined, the data tilt point location is obtained, and the data tilt point location is output.

[0014] Furthermore, the step of obtaining the association relationship of the target object and marking the association relationship in the HQL execution log to obtain the association identifier specifically includes:

[0015] Identify associated objects that are related to the target object;

[0016] Obtain the association relationship between the target object and the associated object;

[0017] The association is searched in the HQL execution log, and the association is marked to obtain the association identifier.

[0018] Furthermore, the step of obtaining the HQL execution log and identifying the target object exhibiting data skew and the skew log field within the HQL execution log specifically includes:

[0019] Obtain the HQL execution log and determine whether there are any target objects in the HQL execution log that exhibit data skew.

[0020] If present, the frequency of occurrence of each key value of the target object is monitored within a preset time period;

[0021] Key values ​​that appear more frequently than a preset threshold are defined as values ​​indicating data skew.

[0022] The skewed log field is obtained by searching the HQL execution log for the log field corresponding to the key value that shows data skew.

[0023] Furthermore, after obtaining the HQL execution log and determining whether there are target objects exhibiting data skew in the HQL execution log, the process also includes:

[0024] If it does not exist, then set global parameters to monitor the HQL execution plan process;

[0025] Specifically, the setting of global parameters to monitor the HQL execution plan process includes:

[0026] The associated execution tasks in the HQL execution plan process are monitored using the global parameters.

[0027] When a related execution task is detected, the frequency of occurrence of each key value of each object in the HQL execution log is monitored within a preset time period.

[0028] Key values ​​that appear more frequently than a preset threshold are defined as values ​​indicating data skew, and the objects corresponding to the key values ​​indicating data skew are identified as the target objects.

[0029] The skewed log field is obtained by searching the HQL execution log for the log field corresponding to the key value that shows data skew.

[0030] Further, determining the location of the difference in the flow relationship based on the association identifier, obtaining the data skew point location, and outputting the data skew point location specifically includes:

[0031] In the phase plan of the HQL execution log, identify the association identifier that is closest to the difference in the flow relationship;

[0032] Obtain the location information of the closest associated identifier in the stage plan of the HQL execution log;

[0033] Based on the location information, the location of the difference in the flow relationship is determined, the location of the data tilt point is obtained, and the location of the data tilt point is output.

[0034] Furthermore, determining the association identifier in the stage plan of the HQL execution log that is closest to the difference in the flow relationship specifically includes:

[0035] Starting from the position of the difference in the flow relationship, traverse the stage plan of the HQL execution log forward and backward in rows respectively;

[0036] Identify the association identifier that is closest to the difference in the flow relationship from the stage plan of the HQL execution log.

[0037] Furthermore, after determining the location of the difference in the flow relationship based on the association identifier, obtaining the data skew point location, and outputting the data skew point location, the method further includes:

[0038] Receive a data anomaly query command and obtain the location of the data tilt point carried in the data anomaly query command;

[0039] Query the data flow relationship corresponding to the data skew point position in the stage plan of the HQL execution log, and output the data flow relationship corresponding to the data skew point position.

[0040] To address the aforementioned technical problems, this application also provides a data tilt point finding device, which employs the following technical solution:

[0041] A data tilt point finding device, comprising:

[0042] The log acquisition module is used to acquire HQL execution logs and determine the target object where data skew occurs and the skew log field in the HQL execution logs.

[0043] The association marking module is used to obtain the association relationship of the target object and mark the association relationship in the HQL execution log to obtain the association identifier;

[0044] The plan parsing module is used to parse the HQL execution plan and determine the first data flow relationship corresponding to the skew log field based on the parsing results;

[0045] The plan acquisition module is used to acquire the stage plan of the HQL execution log and determine the second data flow relationship corresponding to the skew log field from the stage plan;

[0046] The relationship comparison module is used to compare the first data flow relationship and the second data flow relationship to determine the differences in the flow relationship;

[0047] The tilt point identification module is used to determine the position of the difference in the flow relationship based on the association identifier, obtain the data tilt point position, and output the data tilt point position.

[0048] To address the aforementioned technical problems, this application also provides a computer device that employs the following technical solution:

[0049] A computer device includes a memory and a processor, the memory storing computer-readable instructions, the processor executing the computer-readable instructions to implement the steps of the data skew point lookup method as described in any of the preceding claims.

[0050] To address the aforementioned technical problems, this application also provides a computer-readable storage medium, employing the technical solution described below:

[0051] A computer-readable storage medium storing computer-readable instructions, which, when executed by a processor, implement the steps of the data skew point lookup method as described in any one of the preceding descriptions.

[0052] Compared with the prior art, the embodiments of this application have the following main advantages:

[0053] This application discloses a method, apparatus, computer equipment, and storage medium for finding data skew points, belonging to the fields of big data technology and insurance and finance. This application identifies the target object and skew log field exhibiting data skew in the HQL execution log, obtains the association relationship of the target object, marks the association relationship in the HQL execution log to obtain an association identifier, parses the HQL execution plan, determines the first data flow relationship corresponding to the skew log field based on the parsing result, obtains the stage plan of the HQL execution log, determines the second data flow relationship corresponding to the skew log field from the stage plan, compares the first and second data flow relationships to determine the differences in flow relationships, determines the location of the difference in flow relationships based on the association identifier, obtains the location of the data skew point, and outputs the data skew point location. This application locates data skew points by combining data flow relationships and association identifiers. Compared to the traditional method of manually analyzing HQL execution logs, this application can quickly locate data skew points in the database of an insurance and financial management system, reducing manpower input and improving the accuracy of data skew point finding. Attached Figure Description

[0054] To more clearly illustrate the solutions in this application, the accompanying drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0055] Figure 1 An exemplary system architecture diagram is shown, in which this application can be applied;

[0056] Figure 2 A flowchart of one embodiment of the data tilt point finding method according to this application is shown;

[0057] Figure 3 A schematic diagram of one embodiment of the data tilt point finding device according to this application is shown;

[0058] Figure 4 A schematic diagram of the structure of one embodiment of a computer device according to this application is shown. Detailed Implementation

[0059] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein in the specification of the application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application; the terms "comprising" and "having," and any variations thereof, in the specification, claims, and foregoing drawings of this application, are intended to cover non-exclusive inclusion. The terms "first," "second," etc., in the specification, claims, or foregoing drawings of this application are used to distinguish different objects, not to describe a particular order.

[0060] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0061] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

[0062] like Figure 1 As shown, system architecture 100 may include terminal devices 101, 102, and 103, a network 104, and a server 105. Network 104 serves as the medium for providing communication links between terminal devices 101, 102, and 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0063] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, and 103, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social media platform software, etc.

[0064] Terminal devices 101, 102, and 103 can be various electronic devices with displays and support web browsing, including but not limited to smartphones, tablets, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III), MP4 players (Moving Picture Experts Group Audio Layer IV), laptops, and desktop computers, etc.

[0065] Server 105 can be a server that provides various services, such as a backend server that supports the pages displayed on terminal devices 101, 102, and 103. The server can be a standalone server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0066] It should be noted that the data tilt point finding method provided in the embodiments of this application is generally executed by a server, and correspondingly, the data tilt point finding device is generally set in the server.

[0067] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0068] Continue to refer to Figure 2 The diagram illustrates a flowchart of an embodiment of the data tilt point finding method according to this application. Embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that utilize digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0069] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0070] Data skew is a common problem in the big data industry. However, it is usually handled manually. But because HQL logs are generally long and have high logical complexity, it takes a lot of effort to handle data skew manually. Moreover, manual handling is prone to errors and has low accuracy.

[0071] To address the aforementioned technical problems, this application discloses a method, apparatus, computer device, and storage medium for identifying data skew points, belonging to the fields of big data technology and insurance and finance. This application locates data skew points by combining data flow relationships and association identifiers. Compared to the traditional method of manually analyzing HQL execution logs, this application can quickly locate data skew points in the database, reducing manpower input and improving the accuracy of data skew point identification.

[0072] This application's embodiments are specifically designed for optimizing the use of HQL statements in a distributed computing engine to perform join operations on two tables, particularly in cases of data skew. By comparing the data flow relationships corresponding to the HQL execution plan with the data flow relationships of the initial HQL execution log's stage plan, the differences in flow relationships are determined. Combined with the association relationship identifier, the data skew point can be located, which can accelerate the efficiency of data skew point location, reduce manpower input, and improve the accuracy of data skew point finding.

[0073] In this embodiment, HQL is an abbreviation for Hibernate Query Language, providing richer, more flexible, and more powerful query capabilities; HQL is closer to SQL query syntax. Hibernate Query Language (HQL) is an object-oriented query language, similar to SQL, but instead of manipulating tables and columns, it focuses on objects and their attributes. HQL queries are translated by Hibernate into traditional SQL queries to operate on the database.

[0074] The data skew point finding method includes the following steps:

[0075] S201, retrieve the HQL execution log, and identify the target object and skew log field where data skew occurs in the HQL execution log.

[0076] In this embodiment, the server obtains the HQL execution log generated when executing the HQL script, and monitors each object in the HQL execution log according to a preset monitoring method to determine the target object where data skew occurs and the skew log field.

[0077] It should be noted that executing HQL scripts will generate corresponding log information. You can choose to download the log information manually or use Python to automate the process. Using logs to find data skew points is a common practice in the industry.

[0078] Furthermore, obtain the HQL execution logs and identify the target object exhibiting data skew and the skew log fields within the HQL execution logs, specifically including:

[0079] Obtain the HQL execution log and determine if there are any target objects in the HQL execution log that exhibit data skew.

[0080] If it exists, monitor the frequency of occurrence of each key value of the target object within a preset time period;

[0081] Key values ​​that appear more frequently than a preset threshold are defined as values ​​indicating data skew.

[0082] Find the log field corresponding to the key value that shows data skew in the HQL execution log to obtain the skewed log field.

[0083] In this embodiment, by acquiring the HQL execution log and determining whether there is a target object with data skew in the HQL execution log, if it has been determined in advance that there is a target object with data skew in the HQL execution log, the frequency of occurrence of each key value of the target object is monitored within a preset time period. Key values ​​with a frequency exceeding a preset threshold are defined as values ​​with data skew. For example, a threshold of 99% reduction can be set, and a number of times exceeding this threshold is considered as data skew. For example, 5 times or more is considered as data skew. The log field corresponding to the key value with data skew is searched in the HQL execution log to obtain the skew log field.

[0084] Furthermore, after obtaining the HQL execution log and determining whether there are target objects exhibiting data skew in the HQL execution log, the process also includes:

[0085] If it does not exist, set global parameters to monitor the HQL execution plan process;

[0086] This includes setting global parameters to monitor the HQL execution plan process, specifically:

[0087] Monitor associated execution tasks during the HQL execution plan process using global parameters;

[0088] When a related task is detected, the frequency of occurrence of each key value of each object in the HQL execution log is monitored within a preset time period.

[0089] Key values ​​that appear more frequently than a preset threshold are defined as values ​​indicating data skew, and the objects corresponding to the key values ​​indicating data skew are identified as target objects.

[0090] Find the log field corresponding to the key value that shows data skew in the HQL execution log to obtain the skewed log field.

[0091] In this embodiment, the HQL execution log is obtained, and it is determined whether there is a target object with data skew in the HQL execution log. If no target object with data skew is found in the HQL execution log, it is necessary to monitor the associated execution tasks in the HQL execution plan to identify the target object with data skew. The HQL execution plan process is monitored by setting global parameters. When an associated execution task is detected, the frequency of occurrence of each key value of each object in the HQL execution log is monitored within a preset time period. Key values ​​with a frequency exceeding a preset threshold are defined as values ​​with data skew, and the objects corresponding to the key values ​​with data skew are identified as target objects. The log field corresponding to the key value with data skew is searched in the HQL execution log to obtain the skew log field.

[0092] In the above embodiments, when the HQL execution log is generated during the execution of the HQL script, the hints in the HQL statements specify the tables and columns where data skew may occur. If the target object of the data skew is already known, i.e., the table or column where data skew occurs, the log field corresponding to the key value with data skew can be directly found by monitoring the frequency of occurrence of each key value of the target object. However, if no target object with data skew exists, it is necessary to monitor it by setting global parameters to determine the target object with data skew, and only then can the log field corresponding to the key value with data skew be directly found by monitoring the frequency of occurrence of each key value of the target object.

[0093] It should be noted that the global parameter can be set through the skewjoin.autodetect property. When a join operation of two tables or two columns is automatically detected, if the frequency of occurrence of each key value in each object in the distributed partition exceeds a threshold percentage (e.g., 99%) within a preset time period, such as 1 minute, it is determined that the join operation of the associated object has data skew. The preset time period can be set through the skewjoin.observationtime property.

[0094] S202, obtain the association relationship of the target object, and mark the association relationship in the HQL execution log to obtain the association identifier.

[0095] In this embodiment, the server traverses the HQL execution log, searches for all objects in the log, and identifies the relationships between the target object and other objects. These relationships are then marked in the HQL execution log to obtain association identifiers. The association relationship is essentially an ON condition, a fixed syntax in HQL that indicates a relationship between two tables or columns. For example, comparing the ability levels of two people of the same age would involve a height relationship, i.e., ON A's age = B's age. Marking these relationships helps in subsequent data skew point identification.

[0096] Furthermore, the relationships between the target objects are obtained, and these relationships are marked in the HQL execution log to obtain the relationship identifiers, specifically including:

[0097] Identify the related objects that are associated with the target object;

[0098] Obtain the relationship between the target object and related objects;

[0099] Search for relationships in the HQL execution log, mark the relationships, and obtain the relationship identifier.

[0100] In this embodiment, the server traverses the HQL execution log to find associated objects that are related to the target object, obtains the association relationship between the target object and the associated objects, searches for the association relationship in the HQL execution log, marks the association relationship, and obtains the association identifier.

[0101] S203, parse the HQL execution plan and determine the first data flow relationship corresponding to the skew log field based on the parsing results.

[0102] In this embodiment, the server obtains the data flow relationship Stage-N1 corresponding to the HQL execution log by parsing the HQL execution plan. Here, Stage-N represents a specific stage in the HQL execution plan, indicating a certain sequential relationship between stages. Stage-N1 represents the data flow relationship corresponding to the HQL execution log generated when the HQL script is actually executed.

[0103] S204, obtain the stage plan of the HQL execution log, and determine the second data flow relationship corresponding to the skew log field from the stage plan.

[0104] In this embodiment, the server obtains the stage plan (Stage-N0) of the HQL execution log and determines the second data flow relationship corresponding to the skew log field from the stage plan (Stage-N0). The stage plan (Stage-N0) is the stage plan set during HQL script development, which contains the data flow relationship between each execution step.

[0105] S205, compare the first data flow relationship and the second data flow relationship to determine the differences in the flow relationship.

[0106] In this embodiment, by comparing the first data flow relationship Stage-N1 and the second data flow relationship Stage-N0, the difference between the first data flow relationship Stage-N1 and the second data flow relationship Stage-N0 is determined, and the data skew point position is obtained through the position of the flow relationship difference. The data skew point position is marked in the stage plan of the HQL execution log so as to perform data skew processing.

[0107] S206. Based on the association identifier, determine the location of the difference in the flow relationship, obtain the location of the data skew point, and output the location of the data skew point.

[0108] In this embodiment, the location of the difference in the flow relationship in the stage plan of the HQL execution log is determined according to the association identifier, the data skew point location is obtained, and the data skew point location is output.

[0109] Furthermore, based on the association identifier, the location of the difference in the flow relationship is determined, the location of the data skew point is obtained, and the data skew point location is output, specifically including:

[0110] Identify the association identifier that most closely matches the difference in the flow relationship in the phase plan of the HQL execution log;

[0111] Obtain the location information of the closest associated identifier in the stage plan of the HQL execution log;

[0112] Based on location information, the location of the difference in flow relationship is determined, the location of the data tilt point is obtained, and the location of the data tilt point is output.

[0113] In this embodiment, the server determines the association identifier closest to the difference in the flow relationship in the stage plan of the HQL execution log, obtains the position information of the closest association identifier in the stage plan of the HQL execution log, determines the position of the difference in the flow relationship based on the position information, obtains the data skew point position, and outputs the data skew point position.

[0114] Furthermore, the association identifier closest to the difference in the workflow relationship is identified in the stage plan of the HQL execution log, specifically including:

[0115] Starting from the position where the flow relationship differs, traverse the HQL execution log stage plan forward and backward in rows respectively;

[0116] Identify the association identifier that most closely matches the difference in the flow relationship from the stage plan of the HQL execution log.

[0117] In this embodiment, the server traverses the stage plan of the HQL execution log forward and backward in rows, starting from the position of the difference in the flow relationship, and determines the association identifier that is closest to the difference in the flow relationship from the stage plan of the HQL execution log.

[0118] In a specific embodiment of this application, using the `explain formatted+HQL` script is a useful tool for obtaining HQL execution plans. The typical execution flow involves joining tables, almost always with an `ON` condition (e.g., `table1 field = table2 field`). When checking for skew, the table corresponding to the stage in the HQL execution plan will be found. However, if there is no direct table corresponding to the stage, testers will be at a loss. But if the corresponding `ON` conditions are printed from the stage, and all `ON` conditions are marked with special identifiers (e.g., `nvl(table1 field, 'hive1') = table2 field`), and special identifiers are quickly added to all `ON` conditions using Python, and the stage plans of the HQL execution log are traversed forward and backward row by row starting from the `ON` condition identifiers, and denoted as `hive1`, `hive2`, `hive3`, ..., `hive3N`, then the process can be improved. In other words, by marking the ON condition from the association identifier, the skewed stage-N can be directly obtained. Then, the corresponding hiveN in stage-N can be found. By directly comparing it with the marked position in HQ, the corresponding difference can be found and the location of the data skew point can be output, and the cause of the skew can be analyzed in detail.

[0119] Furthermore, after determining the location of the data skew point based on the association identifier to identify the difference in the flow relationship, obtaining the data skew point location, and outputting the data skew point location, the process also includes:

[0120] Receive a data anomaly query command and obtain the location of the data tilt point carried in the data anomaly query command;

[0121] Query the data flow relationship corresponding to the data skew point position in the stage plan of the HQL execution log, and output the data flow relationship corresponding to the data skew point position.

[0122] In this embodiment, when the server receives a data anomaly query command, it obtains the data skew point location carried in the data anomaly query command, queries the data flow relationship corresponding to the data skew point location in the stage plan of the HQL execution log, and outputs the data flow relationship corresponding to the data skew point location for users to perform data skew analysis and processing.

[0123] In this embodiment, the data tilt point finding method operates on an electronic device (e.g., Figure 1 The server shown can receive data anomaly query commands via wired or wireless connection. It should be noted that the aforementioned wireless connection methods may include, but are not limited to, 3G / 4G connections, WiFi connections, Bluetooth connections, WiMAX connections, Zigbee connections, UWB (ultra wideband) connections, and other currently known or future known wireless connection methods.

[0124] In the above embodiments, this application discloses a method for finding data skew points, belonging to the fields of big data technology and insurance and finance. This application identifies the target object and skew log field exhibiting data skew in the HQL execution log, obtains the association relationship of the target object, marks the association relationship in the HQL execution log to obtain an association identifier, parses the HQL execution plan, determines the first data flow relationship corresponding to the skew log field based on the parsing result, obtains the stage plan of the HQL execution log, determines the second data flow relationship corresponding to the skew log field from the stage plan, compares the first and second data flow relationships to determine the difference in flow relationships, determines the location of the difference in flow relationships based on the association identifier, obtains the data skew point location, and outputs the data skew point location. This application locates data skew points by combining data flow relationships and association identifiers. Compared to the traditional method of manually analyzing HQL execution logs, this application can quickly locate data skew points in the database, reducing manpower input and improving the accuracy of data skew point finding.

[0125] In one specific embodiment of this application, taking the data skew point detection in an insurance claims system as an example, the process first involves identifying the target object and skew log field exhibiting data skew in the HQL execution log. Then, the relationships between the target object and these relationships are obtained and marked to obtain a relationship identifier. Next, the first data flow relationship corresponding to the skew log field is determined, the stage plan of the HQL execution log is obtained, and the second data flow relationship corresponding to the skew log field is determined. The first and second data flow relationships are compared to identify differences in the flow relationships. Finally, the location of the difference in the flow relationships is determined based on the relationship identifier to obtain the location of the data skew point. Through this technical solution, the insurance claims system can more quickly detect and resolve data skew issues, improve data processing efficiency, increase claims processing speed, and enhance user experience.

[0126] It should be emphasized that, to further ensure the privacy and security of the HQL execution logs, the HQL execution logs can also be stored in a node of a blockchain.

[0127] The blockchain referred to in this application is a novel application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms. Essentially, a blockchain is a decentralized database, a chain of data blocks linked together using cryptographic methods. Each data block contains information about a batch of network transactions, used to verify the validity of the information (anti-counterfeiting) and generate the next block. A blockchain can include an underlying blockchain platform, a platform product service layer, and an application service layer.

[0128] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by instructing related hardware with computer-readable instructions. These computer-readable instructions can be stored in a computer-readable storage medium, and when executed, they can include the processes of the embodiments of the methods described above. The aforementioned storage medium can be a non-volatile storage medium such as a magnetic disk, optical disk, or read-only memory (ROM), or random access memory (RAM).

[0129] It should be understood that although the steps in the flowcharts of the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0130] Further reference Figure 3 As a response to the above Figure 2 To implement the method shown, this application provides an embodiment of a data tilt point finding device, which is similar to... Figure 2 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0131] like Figure 3 As shown, the data tilt point finding device 300 described in this embodiment includes:

[0132] The log acquisition module 301 is used to acquire HQL execution logs and determine the target object and skew log fields in the HQL execution logs where data skew occurs.

[0133] The association marking module 302 is used to obtain the association relationship of the target object and mark the association relationship in the HQL execution log to obtain the association identifier;

[0134] The plan parsing module 303 is used to parse the HQL execution plan and determine the first data flow relationship corresponding to the skew log field based on the parsing results;

[0135] The plan acquisition module 304 is used to acquire the stage plan of the HQL execution log and determine the second data flow relationship corresponding to the skew log field from the stage plan;

[0136] The relationship comparison module 305 is used to compare the first data flow relationship and the second data flow relationship to determine the differences in the flow relationship;

[0137] The tilt point identification module 306 is used to determine the location of the difference in the flow relationship based on the association identifier, obtain the data tilt point location, and output the data tilt point location.

[0138] Furthermore, the association tagging module 302 specifically includes:

[0139] The object lookup unit is used to identify associated objects that are related to the target object;

[0140] The relationship lookup unit is used to obtain the relationship between the target object and related objects.

[0141] The association tagging unit is used to find associations in the HQL execution log, mark the associations, and obtain the association identifier.

[0142] Furthermore, the log acquisition module 301 specifically includes:

[0143] The object determination unit is used to obtain the HQL execution log and determine whether there is a target object in the HQL execution log that shows data skew.

[0144] The first monitoring unit is used to monitor the frequency of occurrence of each key value of the target object within a preset time period if there is a target object with data skew in the HQL execution log.

[0145] The first definition unit is used to define key values ​​whose frequency of occurrence exceeds a preset threshold as values ​​that indicate data skew.

[0146] The first lookup unit is used to find the log field corresponding to the key value that shows data skew in the HQL execution log, and obtain the skewed log field.

[0147] Furthermore, the log acquisition module 301 also includes:

[0148] The parameter setting unit is used to set global parameters to monitor the HQL execution plan process if there is no target object with data skew in the HQL execution log.

[0149] The associated monitoring unit is used to monitor associated execution tasks in the HQL execution plan process through global parameters;

[0150] The second monitoring unit is used to monitor the frequency of occurrence of each key value of each object in the HQL execution log within a preset time period when an associated execution task is detected.

[0151] The second definition unit is used to define key values ​​that appear more frequently than a preset threshold as values ​​that show data skew, and to determine the objects corresponding to key values ​​that show data skew as target objects;

[0152] The second lookup unit is used to find the log field corresponding to the key value that shows data skew in the HQL execution log, and obtain the skewed log field.

[0153] Furthermore, the tilt point recognition module 306 specifically includes:

[0154] The identifier lookup unit is used to identify the association identifier that is closest to the difference in the flow relationship in the stage plan of the HQL execution log;

[0155] The location acquisition unit is used to obtain the location information of the nearest associated identifier in the stage plan of the HQL execution log;

[0156] The tilt point identification unit is used to determine the location of the difference in the flow relationship based on the location information, obtain the data tilt point location, and output the data tilt point location.

[0157] Furthermore, the identifier lookup unit specifically includes:

[0158] Traverse sub-units, used to traverse the HQL execution log stage plan forward and backward, starting from the position where the flow relationship differs, in row units;

[0159] The identifier lookup subunit is used to determine the association identifier that is closest to the difference in the flow relationship from the stage plan of the HQL execution log.

[0160] Furthermore, the data tilt point finding device 300 also includes:

[0161] The query module is used to receive data anomaly query commands and obtain the location of the data tilt point carried in the data anomaly query command;

[0162] The output module is used to query the data flow relationship corresponding to the data skew point position in the stage plan of the HQL execution log, and output the data flow relationship corresponding to the data skew point position.

[0163] In the above embodiments, this application discloses a data skew point finding device, belonging to the fields of big data technology and insurance and finance. This application identifies the target object and skew log field exhibiting data skew in the HQL execution log, obtains the association relationship of the target object, marks the association relationship in the HQL execution log to obtain an association identifier, parses the HQL execution plan, determines the first data flow relationship corresponding to the skew log field based on the parsing result, obtains the stage plan of the HQL execution log, determines the second data flow relationship corresponding to the skew log field from the stage plan, compares the first and second data flow relationships to determine the difference in flow relationships, determines the location of the difference in flow relationships based on the association identifier, obtains the data skew point location, and outputs the data skew point location. This application locates data skew points by combining data flow relationships and association identifiers. Compared to the traditional method of manually analyzing HQL execution logs, this application can quickly locate data skew points in the database, reducing manpower input and improving the accuracy of data skew point finding.

[0164] To address the aforementioned technical problems, embodiments of this application also provide a computer device. Please refer to [link / reference needed]. Figure 4 , Figure 4This is a basic structural block diagram of the computer device in this embodiment.

[0165] The computer device 4 includes a memory 41, a processor 42, and a network interface 43 that are interconnected via a system bus. It should be noted that only the computer device 4 with components 41-43 is shown in the figure; however, it should be understood that it is not required to implement all the shown components, and more or fewer components can be implemented alternatively. Those skilled in the art will understand that the computer device described here is a device capable of automatically performing numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0166] The computer device can be a desktop computer, laptop, handheld computer, or cloud server, etc. The computer device can interact with the user via a keyboard, mouse, remote control, touchpad, or voice control.

[0167] The memory 41 includes at least one type of readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 41 may be an internal storage unit of the computer device 4, such as the hard disk or memory of the computer device 4. In other embodiments, the memory 41 may also be an external storage device of the computer device 4, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the computer device 4. Of course, the memory 41 may also include both the internal storage unit and its external storage device of the computer device 4. In this embodiment, the memory 41 is typically used to store the operating system and various application software installed on the computer device 4, such as computer-readable instructions for data skew point lookup methods. In addition, the memory 41 can also be used to temporarily store various types of data that have been output or will be output.

[0168] In some embodiments, the processor 42 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip. The processor 42 is typically used to control the overall operation of the computer device 4. In this embodiment, the processor 42 is used to execute computer-readable instructions stored in the memory 41 or to process data, for example, to execute computer-readable instructions for the data skew point lookup method.

[0169] The network interface 43 may include a wireless network interface or a wired network interface, which is typically used to establish communication connections between the computer device 4 and other electronic devices.

[0170] In the above embodiments, this application discloses a computer device belonging to the fields of big data technology and insurance and finance. This application identifies the target object exhibiting data skew and the skew log field in the HQL execution log, obtains the association relationship of the target object, marks the association relationship in the HQL execution log to obtain an association identifier, parses the HQL execution plan, determines the first data flow relationship corresponding to the skew log field based on the parsing result, obtains the stage plan of the HQL execution log, determines the second data flow relationship corresponding to the skew log field from the stage plan, compares the first and second data flow relationships to determine the difference in flow relationships, determines the location of the difference in flow relationships based on the association identifier, obtains the data skew point location, and outputs the data skew point location. This application locates data skew points by combining data flow relationships and association identifiers. Compared to the traditional method of manually analyzing HQL execution logs, this application can quickly locate data skew points in the database, reducing manpower input and improving the accuracy of data skew point finding.

[0171] This application also provides another embodiment, namely, providing a computer-readable storage medium storing computer-readable instructions that can be executed by at least one processor to cause the at least one processor to perform the steps of the data skew point finding method described above.

[0172] In the above embodiments, this application discloses a storage medium belonging to the fields of big data technology and insurance and finance. This application identifies the target object exhibiting data skew and the skew log field in the HQL execution log, obtains the association relationship of the target object, marks the association relationship in the HQL execution log to obtain an association identifier, parses the HQL execution plan, determines the first data flow relationship corresponding to the skew log field based on the parsing result, obtains the stage plan of the HQL execution log, determines the second data flow relationship corresponding to the skew log field from the stage plan, compares the first and second data flow relationships to determine the difference in flow relationships, determines the location of the difference in flow relationships based on the association identifier, obtains the data skew point location, and outputs the data skew point location. This application locates data skew points by combining data flow relationships and association identifiers. Compared to the traditional method of manually analyzing HQL execution logs, this application can quickly locate data skew points in the database, reducing manpower input and improving the accuracy of data skew point finding.

[0173] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0174] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0175] Obviously, the embodiments described above are only some embodiments of this application, not all embodiments. The accompanying drawings show preferred embodiments of this application, but do not limit the patent scope of this application. This application can be implemented in many different forms; rather, the purpose of providing these embodiments is to provide a more thorough and comprehensive understanding of the disclosure of this application. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing specific embodiments, or make equivalent substitutions for some of the technical features. Any equivalent structures made using the content of this application's specification and drawings, directly or indirectly applied to other related technical fields, are similarly within the scope of patent protection of this application.

Claims

1. A method for finding data skew points, characterized in that, include: Obtain the HQL execution log, and identify the target object where data skew occurs and the skew log field in the HQL execution log; Obtain the association relationships of the target object, and mark the association relationships in the HQL execution log to obtain the association identifier; Parse the HQL execution plan and determine the first data flow relationship corresponding to the skew log field based on the parsing results; Obtain the phase plan of the HQL execution log, and determine the second data flow relationship corresponding to the skew log field from the phase plan; Compare the first data flow relationship with the second data flow relationship to determine the differences in the flow relationship; Based on the association identifier, the location of the difference in the flow relationship is determined, the data tilt point location is obtained, and the data tilt point location is output. The step of determining the location of the difference in the flow relationship based on the association identifier, obtaining the data skew point location, and outputting the data skew point location specifically includes: In the phase plan of the HQL execution log, identify the association identifier that is closest to the difference in the flow relationship; Obtain the location information of the closest associated identifier in the stage plan of the HQL execution log; Based on the location information, the location of the difference in the flow relationship is determined, the location of the data tilt point is obtained, and the location of the data tilt point is output.

2. The data tilt point finding method as described in claim 1, characterized in that, The step of obtaining the association relationship of the target object and marking the association relationship in the HQL execution log to obtain the association identifier specifically includes: Identify associated objects that are related to the target object; Obtain the association relationship between the target object and the associated object; The association is searched in the HQL execution log, and the association is marked to obtain the association identifier.

3. The data tilt point finding method as described in claim 1, characterized in that, The step of obtaining the HQL execution log and identifying the target object and skew log fields in the HQL execution log specifically includes: Obtain the HQL execution log and determine whether there are any target objects in the HQL execution log that exhibit data skew. If present, the frequency of occurrence of each key value of the target object is monitored within a preset time period; Key values ​​that appear more frequently than a preset threshold are defined as values ​​indicating data skew. The skewed log field is obtained by searching the HQL execution log for the log field corresponding to the key value that shows data skew.

4. The data tilt point finding method as described in claim 3, characterized in that, After obtaining the HQL execution log and determining whether there are target objects exhibiting data skew in the HQL execution log, the process also includes: If it does not exist, then set global parameters to monitor the HQL execution plan process; Specifically, the setting of global parameters to monitor the HQL execution plan process includes: The associated execution tasks in the HQL execution plan process are monitored using the global parameters. When a related execution task is detected, the frequency of occurrence of each key value of each object in the HQL execution log is monitored within a preset time period. Key values ​​that appear more frequently than a preset threshold are defined as values ​​indicating data skew, and the objects corresponding to the key values ​​indicating data skew are identified as the target objects. The skewed log field is obtained by searching the HQL execution log for the log field corresponding to the key value that shows data skew.

5. The data tilt point finding method as described in claim 1, characterized in that, The step of determining the association identifier in the stage plan of the HQL execution log that is closest to the difference in the flow relationship specifically includes: Starting from the position of the difference in the flow relationship, traverse the stage plan of the HQL execution log forward and backward in rows respectively; Identify the association identifier that is closest to the difference in the flow relationship from the stage plan of the HQL execution log.

6. The data skew point finding method as described in any one of claims 1 to 5, characterized in that, After determining the location of the difference in the flow relationship based on the association identifier, obtaining the data skew point location, and outputting the data skew point location, the method further includes: Receive a data anomaly query command and obtain the location of the data tilt point carried in the data anomaly query command; Query the data flow relationship corresponding to the data skew point position in the stage plan of the HQL execution log, and output the data flow relationship corresponding to the data skew point position.

7. A data tilt point finding device, characterized in that, include: The log acquisition module is used to acquire HQL execution logs and determine the target object where data skew occurs and the skew log field in the HQL execution logs. The association marking module is used to obtain the association relationship of the target object and mark the association relationship in the HQL execution log to obtain the association identifier; The plan parsing module is used to parse the HQL execution plan and determine the first data flow relationship corresponding to the skew log field based on the parsing results; The plan acquisition module is used to acquire the stage plan of the HQL execution log and determine the second data flow relationship corresponding to the skew log field from the stage plan; The relationship comparison module is used to compare the first data flow relationship and the second data flow relationship to determine the differences in the flow relationship; The tilt point identification module is used to determine the position of the difference in the flow relationship based on the association identifier, obtain the data tilt point position, and output the data tilt point position; The tilt point recognition module specifically includes: The identifier lookup unit is used to identify the association identifier that is closest to the difference in the flow relationship in the stage plan of the HQL execution log; The location acquisition unit is used to obtain the location information of the nearest associated identifier in the stage plan of the HQL execution log; The tilt point identification unit is used to determine the location of the difference in the flow relationship based on the location information, obtain the data tilt point location, and output the data tilt point location.

8. A computer device, characterized in that, The device includes a memory and a processor, wherein the memory stores computer-readable instructions, and the processor executes the computer-readable instructions to implement the steps of the data skew point finding method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the data skew point lookup method as described in any one of claims 1 to 6.

Images