Data detection method and device, equipment, storage medium and program product

Abstract

The invention discloses a data detection method and device, equipment, a storage medium and a program product, and relates to the technical field of big data. The method comprises the steps that under the condition that a big data application object runs, running information of at least one target event in the big data application object is acquired; according to the operation information of the at least one target event, the big data application object is detected, a redundancy report of the big data application object is generated, the redundancy report comprises at least one redundancy instance, and the redundancy instance comprises a calling path and a redundancy proportion; the redundancy report is processed, a redundancy information graph of the big data application object is obtained, and the redundancy information graph comprises at least one node and the redundancy proportion and value of each node; and displaying a target node and a target path in the redundant information graph, the target node being a node whose redundancy ratio sum value is greater than a preset threshold in at least one node of the redundant information graph, and the target path being a path associated with the target node.

Description

Technical Field

[0001] The present application belongs to the field of big data technology, and in particular relates to a data detection method, device, equipment, storage medium and program product. Background Art

[0002] With the rapid development of Internet technology and the popularization of terminals such as computers and mobile phones, the Internet has entered thousands of households. Correspondingly, it has also brought about the phenomenon of exponential growth of data, which requires the support of big data applications. Big data applications can achieve efficient management and analysis of large-scale data and are the main infrastructure for many companies to process massive data sets. Therefore, the performance of big data applications is crucial to user experience, so the performance optimization of big data applications has very important research and application value. However, the existing technology treats big data applications as black boxes, making it difficult to locate the root cause of performance problems. Users often need to spend extra effort to manually analyze the root cause of performance problems, and may not be able to fundamentally solve the performance problems of big data applications. Summary of the invention

[0003] The embodiments of the present application provide a data detection method, apparatus, device, storage medium and program product, which can help users fundamentally solve the performance problems of big data applications.

[0004] In a first aspect, an embodiment of the present application provides a data detection method, the method comprising:

[0005] When the big data application object is running, obtaining running information of at least one target event in the big data application object, the running information including precise instruction address, memory access type, memory access address, number of bytes and byte value;

[0006] According to the operation information of the at least one target event, the big data application object is detected to generate a redundancy report of the big data application object, wherein the redundancy report includes at least one redundancy instance, and the redundancy instance includes a call path and a redundancy ratio;

[0007] Processing the redundancy report to obtain a redundancy information graph of the big data application object, the redundancy information graph including at least one node and a redundancy ratio and value of each node, the at least one node being obtained by classifying each calling function in a call tree of the at least one redundant instance, different nodes corresponding to different calling functions, and the redundancy ratio and value being the sum of the redundancy ratios in the calling functions corresponding to the nodes;

[0008] The target node and the target path in the redundant information graph are displayed, wherein the target node is a node in the at least one node of the redundant information graph whose redundancy ratio and value is greater than a preset threshold, and the target path is a path associated with the target node.

[0009] In a second aspect, an embodiment of the present application provides a data detection device, the device comprising:

[0010] an acquisition module, used for acquiring, when the big data application object is running, the running information of at least one target event in the big data application object, the running information including the precise instruction address, the memory access type, the memory access address, the number of bytes and the byte value;

[0011] a detection module, configured to detect the big data application object according to the operation information of the at least one target event, and generate a redundancy report of the big data application object, wherein the redundancy report includes at least one redundancy instance, and the redundancy instance includes a call path and a redundancy ratio;

[0012] A processing module, used for processing the redundancy report to obtain a redundancy information graph of the big data application object, wherein the redundancy information graph includes at least one node and a redundancy ratio and value of each node, wherein the at least one node is obtained by classifying each calling function in a call tree of the at least one redundant instance, and different nodes correspond to different calling functions, and the redundancy ratio and value are the sum of the redundancy ratios in the calling functions corresponding to the nodes;

[0013] A display module is used to display the target node and target path in the redundant information graph, wherein the target node is a node in the at least one node of the redundant information graph whose redundancy ratio and value are greater than a preset threshold, and the target path is a path associated with the target node.

[0014] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a processor and a memory storing computer program instructions; when the processor executes the computer program instructions, the data detection method as described in any one of the above is implemented.

[0015] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which computer program instructions are stored, and when the computer program instructions are executed by a processor, the data detection method as described in any one of the above is implemented.

[0016] In a fifth aspect, an embodiment of the present application provides a computer program product. When instructions in the computer program product are executed by a processor of an electronic device, the electronic device executes any one of the data detection methods described above.

[0017] The data detection method, apparatus, device, storage medium and program product of the embodiments of the present application can obtain the operation information of at least one target event in the big data application object when the big data application object is running, and the operation information includes the precise instruction address, memory access type, memory access address, number of bytes and byte value; and detect the big data application object according to the operation information of at least one target event, and generate a redundancy report of the big data application object, the redundancy report includes at least one redundant instance, and the redundant instance includes a call path and a redundancy ratio; then, the redundancy report is processed to obtain a redundant information graph of the big data application object, the redundant information graph includes at least one node and the redundant ratio and value of each node, at least one node is obtained by classifying each call function in the call tree of at least one redundant instance, different nodes correspond to different call functions, and the redundant ratio and value are the sum of the redundant ratios in the call functions corresponding to the nodes; finally, the target node and the target path in the redundant information graph are displayed, the target node is a node in at least one node of the redundant information graph whose redundant ratio and value are greater than a preset threshold, and the target path is a path associated with the target node. Thus, in an embodiment of the present application, by detecting the operation information of at least one target event in a big data application object, a redundant report including at least one redundant instance can be generated, and a redundant information graph of the big data application object can be generated. Since the redundant information graph includes at least one node and the redundant ratio and value of each node, the target node whose redundant ratio and value is greater than a preset threshold, and the target path associated with the target node, are the root causes of performance problems of the big data application object, thereby helping users to fundamentally solve the performance problems of big data applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solution of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0019] Figure 1 It is a flowchart of a data detection method provided in an embodiment of the present application;

[0020] Figure 2 It is a schematic diagram of the system architecture of a data detection method provided in an embodiment of the present application;

[0021] Figure 3 is a flow chart of another data detection method provided in an embodiment of the present application;

[0022] Figure 4 is a structural schematic diagram of a data detection device provided in an embodiment of the present application;

[0023] Figure 5It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0024] The features and exemplary embodiments of various aspects of the present application will be described in detail below. In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain the present application, rather than to limit the present application. For those skilled in the art, the present application can be implemented without the need for some of these specific details. The following description of the embodiments is only to provide a better understanding of the present application by illustrating the examples of the present application.

[0025] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "include..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0026] With the rapid development of Internet technology and the popularization of terminals such as computers and mobile phones, the Internet has entered thousands of households. Correspondingly, it has also brought about the phenomenon of exponential growth of data, which requires the support of big data applications. Big data applications can achieve efficient management and analysis of large-scale data and are the main infrastructure for many companies to process massive data sets. Therefore, the performance of big data applications is crucial to user experience, so the performance optimization of big data applications has very important research and application value. However, the existing technology treats big data applications as black boxes, making it difficult to locate the root cause of performance problems. Users often need to spend extra effort to manually analyze the root cause of performance problems, and may not be able to fundamentally solve the performance problems of big data applications.

[0027] In order to solve the problems in the prior art, the embodiments of the present application provide a data detection method, device, equipment, storage medium and program product. The data detection method provided by the embodiments of the present application is first introduced below.

[0028] Figure 1 FIG. 1 is a flow chart of a data detection method provided by an embodiment of the present application. Figure 1 As shown, a data detection method may include the following steps S101 to S104:

[0029] S101. When a big data application object is running, obtain running information of at least one target event in the big data application object, where the running information includes an exact instruction address, a memory access type, a memory access address, a byte number, and a byte value;

[0030] S102: Detect the big data application object according to the operation information of at least one target event, and generate a redundancy report of the big data application object, wherein the redundancy report includes at least one redundant instance, and the redundant instance includes a call path and a redundant ratio;

[0031] S103, processing the redundancy report to obtain a redundancy information graph of the big data application object, the redundancy information graph including at least one node and a redundancy ratio and value of each node, the at least one node being obtained by classifying each call function in a call tree of at least one redundant instance, different nodes corresponding to different call functions, and the redundancy ratio and value being the sum of the redundancy ratios in the call functions corresponding to the nodes;

[0032] S104, displaying a target node and a target path in the redundant information graph, wherein the target node is a node in the redundant information graph whose redundancy ratio and value is greater than a preset threshold, and the target path is a path associated with the target node.

[0033] The data detection method of the embodiment of the present application can obtain the operation information of at least one target event in the big data application object when the big data application object is running, and the operation information includes the precise instruction address, memory access type, memory access address, number of bytes and byte value; and detect the big data application object according to the operation information of at least one target event, and generate a redundancy report of the big data application object, the redundancy report includes at least one redundant instance, and the redundant instance includes a call path and a redundancy ratio; then, the redundancy report is processed to obtain a redundant information graph of the big data application object, the redundant information graph includes at least one node and the redundant ratio and value of each node, at least one node is obtained by classifying each call function in the call tree of at least one redundant instance, different nodes correspond to different call functions, and the redundant ratio and value are the sum of the redundant ratios in the call functions corresponding to the nodes; finally, the target node and the target path in the redundant information graph are displayed, the target node is a node in at least one node of the redundant information graph whose redundant ratio and value are greater than a preset threshold, and the target path is a path associated with the target node. Thus, in an embodiment of the present application, by detecting the operation information of at least one target event in a big data application object, a redundant report including at least one redundant instance can be generated, and a redundant information graph of the big data application object can be generated. Since the redundant information graph includes at least one node and the redundant ratio and value of each node, the target node whose redundant ratio and value is greater than a preset threshold, and the target path associated with the target node, are the root causes of performance problems of the big data application object, thereby helping users to fundamentally solve the performance problems of big data applications.

[0034] In S101, the big data application object may be, for example, a big data application platform of Hadoop and Spark. The big data application platforms Hadoop and Spark are widely used in the industry. Relying on the big data platform, big data applications can process massive data quickly.

[0035] The target event may be a precise event in Linux perf, that is, a performance event that can be precisely associated with the instruction at which it occurs.

[0036] The above-mentioned operation information may include precise instruction address, memory access type, memory access address, number of bytes and byte value.

[0037] In the above-mentioned case where the big data application object is running, the running information of at least one target event in the big data application object is obtained. Exemplarily, the environmental parameters of the big data application object can be configured according to the target redundancy type, and the target redundancy type includes any one of a hard write type, a silent write type and a redundant read type; based on the environmental parameters, the big data application object is run; and each target event in a preset target event set is sampled regularly to obtain the running information of each target event, and the target event set includes at least one target event.

[0038] In S102, the redundancy report generally refers to a detailed analysis and summary of the redundancy situation, and may include at least one redundancy instance, and the redundancy instance includes a call path and a redundancy ratio.

[0039] The above-mentioned detecting the big data application object according to the running information of at least one target event and generating a redundancy report of the big data application object may exemplarily include detecting the big data application object according to the running information of at least one target event, identifying at least one target redundant behavior in the big data application object, the target redundant behavior corresponds to the target redundant type, and the target redundant behavior includes one of a hard write behavior, a silent write behavior and a redundant read behavior; constructing a redundant instance corresponding to each target redundant behavior according to the calling path and the redundancy ratio of each target redundant behavior, the calling path is constructed by the source code location information of the target redundant behavior and the corresponding precise instruction address, and the redundancy ratio is the ratio of the number of redundant bytes corresponding to the calling path of the target redundant behavior to the total number of accessed bytes; generating a redundant sub-report of each target redundant type according to at least one redundant instance of each target redundant type; generating a redundant report of the big data application object according to the redundant sub-report of the hard write type, the redundant sub-report of the silent write type and the redundant sub-report of the redundant read type.

[0040] In S103, the redundant information graph generally refers to a graph used to describe the information redundancy in a system in information theory and system science. The redundant information graph may include at least one node and a redundant ratio and value of each node, wherein at least one node is obtained by classifying each call function in a call tree of at least one redundant instance, different nodes correspond to different call functions, and the redundant ratio and value is the sum of the redundant ratios in the call functions corresponding to the node.

[0041] The redundant report is processed as described above to obtain a redundant information graph of the big data application object. Exemplarily, each calling function in the call tree of at least one redundant instance in the redundant report may be classified to obtain at least one node of each redundant instance, where different nodes correspond to different calling functions; at least one redundant instance is merged according to the same node, and the redundant ratio and value of each node are summed up according to the redundant ratio of the calling function corresponding to each node; the redundant information graph of the big data application object is obtained according to the at least one merged node, the redundant ratio and value of each node, and the calling relationship between the calling functions in the call tree of at least one redundant instance.

[0042] In S104, the target node is a node whose redundancy ratio and value is greater than a preset threshold among at least one node in the redundant information graph.

[0043] The above target path is a path associated with the target node.

[0044] The above-mentioned display of the target node and the target path in the redundant information graph may, exemplarily, be that in at least one node of the redundant information graph, a node having a redundancy ratio and value greater than a preset threshold is determined as a target node; a node whose call relationship with the target node is extended is determined as an extended node; a target path of the target node is constructed based on the extended node; the target node and the target path are marked in the redundant information graph, and the redundant information graph is displayed.

[0045] In some embodiments, the above S101 may specifically include:

[0046] Configure the environment parameters of the big data application object according to the target redundancy type, where the target redundancy type includes any one of a hard-write type, a silent write type, and a redundant read type;

[0047] Run big data application objects based on environmental parameters;

[0048] Each target event in a preset target event set is sampled periodically to obtain operation information of each target event, and the target event set includes at least one target event.

[0049] The target redundancy type can be any one of the following: hard-write type, silent write type, and redundant read type. Hard-write: Operation S1 and Operation S2 are two consecutive memory write operations to the memory address M (operation S1 occurs before operation S2). If the program does not read the content at the memory address M between operation S1 and operation S2, operation S1 is called a hard-write. In this case, it is called<S1,S2> is a pair of dead write operations. Silent write: Operation S1 and operation S2 are two consecutive memory write operations to memory address M (operation S1 occurs before operation S2). If the values ​​written to memory address M by operation S1 and operation S2 are exactly the same, operation S1 is called a silent write. In this case, it is called<S1,S2> is a pair of silent write operations. Redundant read: Operation L1 and operation L2 are two consecutive memory read operations on memory address M (operation L1 occurs before operation L2). If the values ​​read from memory address M by operation L1 and operation L2 are exactly the same, operation L2 is called redundant read. In this case, it is called<L1,L2> is a pair of redundant read operations.

[0050] The above target event set may include at least one target event.

[0051] In an embodiment of the present application, the environmental parameters of the big data application object can be configured by a target redundancy type, and the target redundancy type includes any one of a hard-write type, a silent write type, and a redundant read type; then based on the environmental parameters, the big data application object is run; finally, each target event in a preset target event set is sampled regularly to obtain the running information of each target event. Since precise events are sampled at regular intervals and there is no need to monitor events all the time, not only is the operation of the big data application object less disturbed, but precise event sampling can effectively reduce the amount of data collected and reduce the overhead of the big data application object.

[0052] In some embodiments, the above S102 may specifically include:

[0053] Detecting the big data application object according to the operation information of at least one target event, identifying at least one target redundant behavior in the big data application object, where the target redundant behavior corresponds to a target redundant type, and the target redundant behavior includes one of a dead write behavior, a silent write behavior, and a redundant read behavior;

[0054] According to the calling path and redundancy ratio of each target redundant behavior, the redundant instance corresponding to each target redundant behavior is constructed. The calling path is constructed by the source code location information of the target redundant behavior and the corresponding precise instruction address. The redundancy ratio is the ratio of the number of redundant bytes corresponding to the calling path of the target redundant behavior to the total number of accessed bytes.

[0055] generating a redundancy sub-report for each target redundancy type according to at least one redundancy instance of each target redundancy type;

[0056] A redundant report of a big data application object is generated according to the redundant sub-report of a dead-write type, the redundant sub-report of a silent write type, and the redundant sub-report of a redundant read type.

[0057] The target redundancy behavior corresponds to a target redundancy type, and the target redundancy behavior may include one of a dead write behavior, a silent write behavior, and a redundant read behavior.

[0058] The above call path is constructed by the source code location information of the target redundant behavior and the corresponding precise instruction address.

[0059] The above redundancy ratio is the ratio of the number of redundant bytes corresponding to the call path of the target redundant behavior to the total number of accessed bytes.

[0060] In an embodiment of the present application, at least one target redundant behavior in a big data application object is identified through the running information of at least one target event, and a redundant instance corresponding to each target redundant behavior is constructed based on the calling path and redundant ratio of each target redundant behavior. Then, based on at least one redundant instance of each target redundant type, a redundant sub-report of each target redundant type is generated. Finally, different types of redundant sub-reports are merged to generate a redundant report of the big data application object, which provides a comprehensive perspective to view the redundancy status of the entire system or application and helps to understand the global impact of redundant data.

[0061] In some embodiments, the redundant instances corresponding to the target redundant behaviors are constructed based on the calling paths and redundant ratios of the target redundant behaviors, which may specifically include:

[0062] Obtain the source code location information of each target redundant behavior;

[0063] According to the source code location information of each target redundant behavior and the corresponding precise instruction address, the calling path of each target redundant behavior is constructed;

[0064] Calculate the redundancy ratio of each target redundant behavior according to the number of redundant bytes and the total number of access bytes corresponding to the call path of each target redundant behavior;

[0065] According to the calling paths and redundancy ratios of each target redundant behavior, redundant instances corresponding to each target redundant behavior are constructed.

[0066] The above source code location information may refer to the specific location of the code in the source code file, which includes the line number and sometimes the column number. In programming, source code location information is very important because it can help developers accurately locate the specific location in the code, especially when debugging a program or reading the code.

[0067] The above-mentioned acquisition of the source code location information of each target redundant behavior can be, for example, obtained by using the application programming interface (Application Programming Interface, API) provided by the Java virtual machine (Java Virtual Machine, JVM) to obtain the function call context, establish a mapping relationship between the redundant operation and the source code location, and thus provide the user with intuitive redundant operation information. The value of the program counter when the sampling is triggered during the program execution can be obtained by using the performance monitoring unit (Performance Monitoring Unit, PMU). Since JVM is a language based on a virtual machine, it is impossible to directly establish a mapping relationship between the instruction address and the source code. The program context obtained by the PMU can be passed to the API AsyncGetCallTrace provided by the JVM, so as to obtain the method ID and bytecode index of each stack frame in the call context of the Java program when the sampling is triggered, and then obtain the call context at the JVM level. Further, the JVM provides the Java Virtual Machine Tool Interface (Java Virtual Machine Tool Interface, JVMTI) APIGetLineNumberTable, which maintains a line number table and can establish a mapping relationship between the bytecode index and the line number. Therefore, the source code location information of the actual execution of the program when the sampling occurs can be obtained by looking up the table.

[0068] The above-mentioned redundant ratio of each target redundant behavior is calculated according to the number of redundant bytes and the total number of access bytes corresponding to the call path of each target redundant behavior. Exemplarily, the redundant ratio of each target redundant behavior can be calculated according to the following formula based on the number of redundant bytes and the total number of access bytes corresponding to the call path of each target redundant behavior. The formula is as follows:

[0069]

[0070] in, The number of redundant bytes corresponding to the call path that constitutes redundant and inefficient behavior; The total number of bytes accessed corresponding to the call path that constitutes redundant and inefficient behavior is calculated by identifying redundant memory operations, including silent writes, dead writes, and redundant reads, in the memory access events that trigger PMU precise event sampling. Specifically, the memory address that triggers PMU sampling is recorded, and the call context information is obtained through JVMTI. <C x ,C y >, calculate the number of redundant memory bytes accessed under a specific call context Count the number of memory bytes accessed by all PMU events And calculate the redundancy ratio

[0071] In the embodiment of the present application, the call path of each target redundant behavior is constructed based on the source code location information and the corresponding precise instruction address of each target redundant behavior, which means that it is possible to determine which parts in the source code are redundant or repeated. Using this information, combined with the precise instruction address, a call path can be constructed to understand how redundant behavior occurs during the execution of the program. According to the number of redundant bytes and the total number of access bytes corresponding to the call path of each target redundant behavior, the redundancy ratio of each target redundant behavior is calculated, providing a quantitative indicator to measure the degree of redundancy, which helps to understand the impact of redundant behavior on program performance and resource consumption. Finally, according to the call path and redundancy ratio of each target redundant behavior, the redundant instance corresponding to each target redundant behavior is constructed to provide strong support for the optimization and maintenance of the program.

[0072] In some embodiments, the above calculation of the redundancy ratio of each target redundant behavior according to the number of redundant bytes and the total number of access bytes corresponding to the call path of each target redundant behavior may specifically include:

[0073] When the target redundancy type is a hard-write type and the target redundancy behavior is a hard-write behavior, the redundancy ratio of each hard-write behavior is calculated according to the number of redundant bytes corresponding to the call path of each hard-write behavior and the total number of written bytes;

[0074] When the target redundancy type is the silent write type and the target redundancy behavior is the silent write behavior, the redundancy ratio of each silent write behavior is calculated according to the number of redundant bytes corresponding to the call path of each silent write behavior and the total number of written bytes;

[0075] When the target redundancy type is a redundant read type and the target redundant behavior is a redundant read behavior, the redundancy ratio of each redundant read behavior is calculated according to the number of redundant bytes corresponding to the call path of each redundant read behavior and the total number of read bytes.

[0076] The redundancy ratio of each dead-write behavior is calculated based on the number of redundant bytes and the total number of written bytes corresponding to the call path of each dead-write behavior. For example, the calculation formula for the dead-write inefficient behavior is as follows:

[0077]

[0078] in, Indicates the number of redundant bytes corresponding to the call path that constitutes the dead-write inefficient behavior, Indicates call context information <C x ,C y >The total number of bytes written to memory.

[0079] The redundancy ratio of each silent write behavior is calculated based on the number of redundant bytes and the total number of written bytes corresponding to the call path of each silent write behavior. For example, the calculation formula for the silent write inefficient behavior is as follows:

[0080]

[0081] in, Indicates the number of redundant bytes corresponding to the call path that constitutes the silent write inefficiency behavior, Indicates call context information <C x ,C y >The total number of bytes written to memory.

[0082] The redundant ratio of each redundant read behavior is calculated based on the number of redundant bytes and the total number of bytes read corresponding to the call path of each redundant read behavior. For example, the calculation formula for the redundant read inefficient behavior is as follows:

[0083]

[0084] in, Indicates the number of redundant bytes corresponding to the call path that constitutes the silent write inefficiency behavior, Indicates call context information <C x ,C y >The total number of bytes read from memory.

[0085] In an embodiment of the present application, the redundancy ratio of each dead write behavior, silent write behavior and redundant read can be calculated according to the calculation formulas corresponding to different target redundancy types and different target redundant behaviors, thereby meeting the redundancy ratio calculation requirements of different target redundant behaviors.

[0086] In some embodiments, the above S103 may specifically include:

[0087] Classify each calling function in the calling tree of at least one redundant instance in the redundant report to obtain at least one node of each redundant instance, where different nodes correspond to different calling functions;

[0088] Merge at least one redundant instance according to the same node, and sum the redundant ratios and values ​​of each node according to the redundant ratios of the corresponding calling functions of each node;

[0089] A redundant information graph of the big data application object is obtained according to at least one merged node, the redundancy ratio and value of each node, and the calling relationship between each calling function in the calling tree of at least one redundant instance.

[0090] The call tree of the redundant instance mentioned above may include at least one calling function. Call Tree: It is a visualization tool used to show the order and hierarchy of function calls during program execution. Each node of the tree represents a function call, and the root node is usually the entry point of the program, such as the main function. Calling function: In the call tree, the function of each node represents a calling action, that is, one function (caller) calls another function (callee). Call tree of redundant instance: specifically refers to the tree representation of all function calls during the execution of redundant instances of the program. This tree shows how redundant behavior occurs through a series of function calls. The above different nodes correspond to different calling functions.

[0091] In an embodiment of the present application, each calling function in a call tree of at least one redundant instance in a redundant report is classified to obtain at least one node of each redundant instance, and then at least one redundant instance is merged according to the same node, and the redundant ratio and value of each node are summed up according to the redundant ratio of the calling function corresponding to each node. Finally, based on the at least one merged node, the redundant ratio and value of each node, and the calling relationship between each calling function in the call tree of at least one redundant instance, a redundant information graph of the big data application object is uniformly constructed, which can help understand the global impact of redundant behavior in the big data application object, thereby providing strong support for program optimization and maintenance, and further realizing the reasonable allocation of resources and the improvement of system performance.

[0092] In some embodiments, the above S104 may specifically include:

[0093] In at least one node of the redundant information graph, a node having a redundancy ratio and value greater than a preset threshold is determined as a target node;

[0094] Determine a node whose call relationship with the target node is extended as an extended node;

[0095] Construct a target path to a target node based on the extended nodes;

[0096] Mark the target node and target path in the redundant information graph and display the redundant information graph.

[0097] The above preset threshold value can be set according to the actual needs of the user and is not limited to a fixed value, so no specific limitation is made here.

[0098] The extended node is a node whose calling relationship with the target node is extended.

[0099] In an embodiment of the present application, in at least one node of the redundant information graph, a node whose redundancy ratio and value is greater than a preset threshold is determined as a target node, and a node whose call relationship with the target node is extended is determined as an extended node, and a target path of the target node is constructed based on the extended node. Finally, the target node and the target path are marked in the redundant information graph, and the redundant information graph is displayed, wherein redundancy elimination optimization on the target node can bring higher performance improvement benefits, and the extended node provides the deep-seated reasons for the occurrence and propagation of redundancy, thereby helping users determine the root cause of redundancy and perform targeted optimization.

[0100] In order to facilitate the understanding of the data detection method in the embodiment of the present application, the actual application process of this data detection method is described as follows:

[0101] In order to overcome the shortcomings of the prior art, the present application provides a data detection method for the redundant and inefficient behaviors existing in applications on widely used big data application platforms such as Hadoop and Spark (equivalent to the above-mentioned big data application objects), including silent writes, dead writes and redundant reads.

[0102] Big data application platforms Hadoop and Spark are widely used in the industry. Relying on the big data platform, big data applications can process massive amounts of data quickly. The performance of big data platforms and framework components greatly affects the execution time of big data tasks, which increases the overhead of big data applications. When programmers use complex and diverse big data application components to write big data processing application codes suitable for different task scenarios, they are prone to introduce redundant and inefficient behaviors due to bad programming habits or long-term code maintenance version iteration legacy issues, which leads to low execution efficiency of the entire big data application. Detecting these redundant and inefficient behaviors helps improve the performance of big data applications. In order to optimize the performance of big data platforms and big data applications, developers need to instrument big data applications and collect performance trace data. However, most of the existing research is coarse-grained analysis, which only treats Hadoop big data applications as black boxes, making it difficult to locate the root cause of performance problems. Therefore, the optimization solutions given based on this often require users to spend extra effort to manually analyze the root cause of performance problems. The proposed detection method for redundant and inefficient behaviors in big data applications uses an Agent mechanism to instrument the system, registers Linux Perf precise event sampling events, collects memory events and their call context information, and finds the redundant and inefficient behaviors that have the greatest impact on performance and are worth optimizing in the big data application source code based on the statistical frequency of redundant occurrence. This method can accurately locate the root cause of performance problems in Hadoop big data applications according to the type of redundancy, and provide users with intuitive optimization guidance.

[0103] The basic idea of ​​this application is to obtain the memory access pattern in the big data application based on sampling, and analyze the memory access pattern to obtain the inefficient instruction behavior in the program, so as to find the root cause of the inefficient bottleneck and guide users to optimize.

[0104] Figure 2 A schematic diagram of the system architecture for implementing a data detection method provided in this application. Using the Java and JVMTI Agent provided by the JVM, the corresponding callback function can be registered for the Hadoop application to listen to the corresponding events, and relevant information can be collected at runtime to detect inefficient behaviors and obtain redundant examples. The redundant report merging module merges the collected multiple redundant examples, and then gives the merged overall redundancy report of the big data application to the visualization module, which is responsible for showing the user the location and proportion of key inefficient behaviors.

[0105] Figure 3 This is a flow chart of another data detection method provided by this application, and the detailed process includes steps 1 to 7:

[0106] Step 1: Register the callback function using Java and JVMTI Agent.

[0107] This application is based on Java and JVMTI Agent registration callback functions to realize the function of collecting runtime information of big data applications such as Hadoop or Spark. This application uses the environment variables HADOOP_OPTS or SPARK_OPTS set in advance before running the Hadoop or Spark application to specify the -javaagent and -agentpath parameters of the JVM at runtime, so that the JVM loads Java and JVMTI Agent, that is, the C / C++ code of the JVMTI Agent is compiled into a shared library for collecting runtime performance data of big data Java applications, and the loading of the Agent is realized. This application packages the JVMTIAgent in libagent.so and uses libagent.so to provide the Agent function. When running a big data Java application, the JVMTI Agent will enter the Agent_OnLoad function and use the interface provided by JVMTI to interact with the JVM, so as to achieve the purpose of collecting performance data of big data Java applications.

[0108] This application provides several sets of pre-configured parameter options, which can be configured in the set_env provided in this application, so that before running Hadoop or Spark big data applications, the environment variables HADOOP_OPTS or SPARK_OPTS are set in advance. In addition to specifying the JVM parameters -javaagent and -agentpath to load the Agent to run, the Agent parameters are also passed in to specify the redundancy detection type. When the application is running, the JVMTI Agent interacts with the JVM, and the Agent_OnLoad function enters the JVM::init function, and then parses the incoming command line parameters, and configures the corresponding callback function for the corresponding event according to the redundancy detection type specified in the Agent parameter using the callback function interface provided by JVMTI, and collects specific runtime information of Hadoop or Spark big data applications. These callback functions are also encapsulated in libagent.so. The callback function registered by the JVMTI Agent will be called when a specific event occurs.

[0109] The callback function registered by the JVMTI Agent will be called when a specific event occurs. The registered callback events and their functions are shown in Table 1.

[0110] Table 1 JVMTI registration callback events

[0111]

[0112]

[0113] Step 2: Register the callback function and collect relevant information.

[0114] Use JVMTI to register a callback function for the event, so that when the thread starts executing, the corresponding callback function is triggered and Linux perf precise event (equivalent to the above target event) sampling is registered, and the memory access event is accurately sampled to obtain runtime information such as precise instruction address, memory access type and address, number of bytes and value (equivalent to the above runtime information);

[0115] Specifically, you can use JVMTI to register JVMTI_EVENT_THREAD_START event monitoring, and trigger the callback function callbackThreadStart when the thread starts executing. The callback function will further call Profiler::getProfiler().threadStart function, thereby using the perf event interface to register Linux perf precise event sampling, and accurately sample memory access events with a sampling interval of 10000, to obtain runtime information such as precise instruction address, memory access type and address, number of bytes and value.

[0116] PMU sampling is used to obtain information related to program reading or writing memory, and further detect three inefficient behaviors: silent write, dead write, and redundant read (equivalent to the target redundant behavior mentioned above). The three inefficient behaviors are defined as follows:

[0117] Dead write: Operation S1 and operation S2 are two consecutive memory write operations to memory address M (operation S1 occurs before operation S2). If the program does not read the content at memory address M between operation S1 and operation S2, operation S1 is called a dead write. In this case, it is called<S1,S2> It is a pair of dead write operations.

[0118] Silent write: Operation S1 and operation S2 are two consecutive memory write operations to memory address M (operation S1 occurs before operation S2). If the values ​​written to memory address M by operation S1 and operation S2 are exactly the same, operation S1 is called a silent write. In this case, it is called<S1,S2> For a pair of silent write operations.

[0119] Redundant read: Operation L1 and operation L2 are two consecutive memory read operations on memory address M (operation L1 occurs before operation L2). If the values ​​read from memory address M by operation L1 and operation L2 are exactly the same, operation L2 is called redundant read. In this case, it is called<L1,L2> is a pair of redundant read operations.

[0120] By using the value information of continuous read and write operations on the same memory address during the execution of the program obtained by sampling, the above three redundant operations of dead write, silent write and redundant read in the program can be detected. Specifically, during the operation of the application under test, each sample sampled by Linux Perf will be recorded in a pre-allocated and specified buffer. After reading the sample from the buffer, the memory address M accessed by the instruction that triggered the sampling, the read / write type and the value v1 stored in the accessed memory address M can be further obtained, and then a hardware interrupt point can be registered for this memory address M. When this address M is accessed again next time, a hardware interrupt is triggered, and the value v2 read / written for the second time for this address M can be obtained. Finally, v1 and v2 are compared according to the definition of the three redundant operations to detect whether there is a corresponding redundant operation for this memory address.

[0121] The API provided by the JVM is used to obtain the function call context, and a mapping relationship is established between the redundant operations and the source code location, thereby providing users with intuitive redundant operation information. The PMU can be used to obtain the value of the program counter when sampling is triggered during program execution. Since the JVM is a language based on a virtual machine, it is impossible to directly establish a mapping relationship between the instruction address and the source code. The program context obtained by the PMU can be passed to the API AsyncGetCallTrace provided by the JVM to obtain the method ID and bytecode index of each stack frame in the call context of the Java program when the sampling is triggered, and then obtain the call context at the JVM level. Furthermore, the JVM provides the JVMTI API GetLineNumberTable, which maintains a line number table and can establish a mapping relationship between the bytecode index and the line number. Therefore, the source code location of the actual execution of the program when sampling occurs (equivalent to the above-mentioned source code location information) can be obtained by looking up the table.

[0122] Step 3: Calculate the redundancy ratio according to the collected runtime information.

[0123] The redundancy ratio is calculated based on the sample, and the line number information (i.e., source code location) is obtained using JVMTI. The call path is constructed in combination with the precise instruction address, and the call path is associated with the redundancy ratio to form a redundant instance.

[0124] In the redundant and inefficient behavior detection of big data applications, the redundancy ratio is calculated according to the following redundancy ratio calculation formula (1):

[0125]

[0126] in, The number of redundant bytes corresponding to the call path that constitutes redundant and inefficient behavior; The total number of bytes accessed corresponding to the call path that constitutes redundant and inefficient behavior is calculated by identifying redundant memory operations, including silent writes, dead writes, and redundant reads, in the memory access events that trigger PMU precise event sampling. Specifically, the memory address that triggers PMU sampling is recorded, and the call context information is obtained through JVMTI. <C x ,C y >, calculate the number of redundant memory bytes accessed under a specific call context Count the number of memory bytes accessed by all PMU events And calculate the redundancy ratio

[0127] The calculation formula (2) for silent write inefficiency behavior is as follows:

[0128]

[0129] in, Indicates the number of redundant bytes corresponding to the call path that constitutes the silent write inefficiency behavior, Indicates call context information <C x ,C y >The total number of bytes written to the memory;

[0130] The calculation formula (3) for the dead-write inefficiency behavior is as follows:

[0131]

[0132] in, Indicates the number of redundant bytes corresponding to the call path that constitutes the dead-write inefficient behavior, Indicates call context information <C x ,C y >The total number of bytes written to the memory;

[0133] The calculation formula (4) for redundant read inefficiency behavior is as follows:

[0134]

[0135] in, Indicates the number of redundant bytes corresponding to the call path that constitutes the redundant read inefficient behavior. Indicates call context information <C x ,C y >The total number of bytes read from memory.

[0136] Step 4: Perform detection and analysis on each redundancy type.

[0137] Take steps 1 to 3 as a one-time process, configure different environment variables HADOOP_OPTS or SPARK_OPTS for different redundancy types, pass different parameters to JVMTI Agent, configure different callback functions for different events for different redundancy types, and perform analysis and diagnosis of various redundant behaviors. Specifically, it is necessary to select the redundancy type required by the user from the pre-configured multiple parameter combinations, run the tool multiple times with these parameters, extract different redundancy analyses in different executions, and obtain a comprehensive redundancy report of the application.

[0138] Step 5: Save the redundancy-related runtime data and generate a redundancy report (equivalent to the above-mentioned redundancy sub-report).

[0139] Using the callback function triggered by the thread end registered by JVMTI, each thread unregisters the Linux perf event and sorts it from high to low according to the redundancy ratio, and prints a redundant report; each thread sorts and outputs the redundant ratio of the redundant instances detected by each thread in the JVMTI callback function triggered by the thread end event. Specifically, this application will store the redundant data of different threads in different files, each file is in XML format, and the redundant information recorded each time is directly stored in sequence in the XML file. In addition, an XML file agent-statistics.run containing overall statistical information will be recorded to record the redundant data common to each thread, as well as a file agent-trace-method.run that records the method name, which is used to reduce the character strings in the redundant report and compress the redundant report.

[0140] Step 6: Merge and compress the redundant reports to generate an overall redundant report of the application (equivalent to the above redundant report).

[0141] This application provides a python2 script that can read the XML redundancy report of each thread, and merge the redundant reports based on this, combine the statistical information agent-statistics.run, use the call path information of the redundant instances of different threads as an index, calculate the redundancy ratio of the redundant instances in the overall operation of the application, and generate the overall redundancy report of the merged application; the redundant instances in the redundant reports printed by each thread contain the corresponding call context path information indicating the location of the redundancy and the redundancy ratio. Since different threads of the same program will execute similar call paths, the redundant instances of different threads can be merged according to the call context path information, and the redundancy ratio of the redundant instances in the overall application can be calculated. The script will also restore the call path of each redundant data based on agent-trace-method.run, and finally generate a redundant report agent-data.

[0142] Step 7: Visualize the redundancy report to guide users to perform targeted optimization.

[0143] This application provides a python3 script for traversing each instance of the overall redundancy report of the application, organizing the redundancy information graph with each function call on the call path as a node and the call relationship as an edge, visualizing the program redundancy situation, and identifying the key nodes and their call paths with a redundancy ratio greater than a predetermined threshold, guiding users to perform targeted optimization. Specifically, it includes the following four sub-steps (7-1) to (7-4):

[0144] (7-1) Read the overall redundant report agent-data of the merged big data application, and split it based on the line break of the string, and parse each redundant instance for each part of the split, including the call path and the redundancy ratio, so that all redundant data instances form a linear data structure;

[0145] (7-2) Analyze the redundant instances of the linear data structure. Each redundant instance has its call tree. Classify each function in the call tree as a node, merge the same functions in different call instances, and sum the redundancy ratios of the same function to obtain a redundant information graph. This application uses the dictionary provided by Python to classify each redundant instance according to its call path, and traverses the graph after the graph structure is built to sum the redundancy ratios of the redundant instances corresponding to the same function calls. The redundant information graph is organized with the function calls on the call path as nodes and the call relationships of different functions on the path as edges. The nodes contain the redundant ratio F_node (equivalent to the above-mentioned redundant ratio and value) accumulated by all redundant instances in this function. The redundancy ratio of the root node of the redundant information graph is 100%, and the redundant ratio will be dispersed to other nodes in the graph layer by layer according to the call path relationship;

[0146] (7-3) Based on the breadth-first traversal algorithm, all nodes in the redundant information graph are traversed to determine the redundancy ratio F_node of all nodes. If the redundancy ratio F_node exceeds the specified threshold, it is marked as a critical redundancy occurrence node (equivalent to the above-mentioned target node), and the nodes extended from the deepest critical redundancy occurrence node according to the call relationship are also marked as critical redundancy extension nodes (equivalent to the above-mentioned extension nodes), thereby constructing a redundant critical path (equivalent to the above-mentioned target path);

[0147] (7-4) Report the redundant critical path to the user. Redundancy elimination optimization on the critical redundancy occurrence nodes on the path can bring higher performance improvement benefits. The critical redundancy extension nodes provide the deep-seated reasons for the occurrence and propagation of redundancy, which helps users determine the root cause of redundancy and perform targeted optimization. Use the graphviz library to convert the redundant information graph into a picture format, and use special colors to mark the critical redundancy occurrence nodes and redundant critical paths to show the user the visual results.

[0148] In the embodiments of the present application, redundant information of applications is collected with low overhead, which can be applied to complex production environments to provide optimization suggestions and guidance for applications in actual production environments. The present application can analyze three types of inefficient behaviors: dead writes, silent writes, and redundant reads, covering a wide range of performance issues, and accurately locate performance bottlenecks in big data applications, providing users with targeted optimization suggestions. The present application performs performance analysis and positioning of big data applications at the instruction-level granularity. The present application is a data detection method with a finer granularity that can be analyzed in-depth within the system.

[0149] Based on the data detection method provided in the above embodiment, the present application also provides a specific implementation of a data detection device. Please refer to the following embodiment.

[0150] like Figure 4 As shown, the data detection device 400 provided in the embodiment of the present application may include the following modules: an acquisition module 401 , a detection module 402 , a processing module 403 and a display module 404 .

[0151] The acquisition module 401 is used to acquire the operation information of at least one target event in the big data application object when the big data application object is running, and the operation information includes the precise instruction address, the memory access type, the memory access address, the number of bytes and the byte value;

[0152] A detection module 402 is used to detect the big data application object according to the operation information of at least one target event, and generate a redundancy report of the big data application object, wherein the redundancy report includes at least one redundant instance, and the redundant instance includes a call path and a redundant ratio;

[0153] The processing module 403 is used to process the redundancy report to obtain a redundancy information graph of the big data application object, wherein the redundancy information graph includes at least one node and a redundancy ratio and value of each node, wherein the at least one node is obtained by classifying each calling function in a calling tree of at least one redundant instance, wherein different nodes correspond to different calling functions, and the redundancy ratio and value are the sum of the redundancy ratios in the calling functions corresponding to the nodes;

[0154] The display module 404 is used to display the target node and the target path in the redundant information graph. The target node is a node in the redundant information graph whose redundancy ratio and value are greater than a preset threshold, and the target path is a path associated with the target node.

[0155] The data detection device of the embodiment of the present application can obtain the operation information of at least one target event in the big data application object when the big data application object is running, and the operation information includes the precise instruction address, memory access type, memory access address, number of bytes and byte value; and detect the big data application object according to the operation information of at least one target event, and generate a redundancy report of the big data application object, the redundancy report includes at least one redundant instance, and the redundant instance includes a call path and a redundancy ratio; then, the redundancy report is processed to obtain a redundant information graph of the big data application object, the redundant information graph includes at least one node and the redundant ratio and value of each node, at least one node is obtained by classifying each call function in the call tree of at least one redundant instance, different nodes correspond to different call functions, and the redundant ratio and value are the sum of the redundant ratios in the call function corresponding to the node; finally, the target node and the target path in the redundant information graph are displayed, the target node is a node in at least one node of the redundant information graph whose redundant ratio and value are greater than a preset threshold, and the target path is a path associated with the target node. Thus, in an embodiment of the present application, by detecting the operation information of at least one target event in a big data application object, a redundant report including at least one redundant instance can be generated, and a redundant information graph of the big data application object can be generated. Since the redundant information graph includes at least one node and the redundant ratio and value of each node, the target node whose redundant ratio and value is greater than a preset threshold, and the target path associated with the target node, are the root causes of performance problems of the big data application object, thereby helping users to fundamentally solve the performance problems of big data applications.

[0156] In some embodiments, the acquisition module 401 may specifically include:

[0157] A configuration submodule, used to configure the environment parameters of the big data application object according to the target redundancy type, where the target redundancy type includes any one of a hard-write type, a silent write type, and a redundant read type;

[0158] The running submodule is used to run big data application objects based on environmental parameters;

[0159] The acquisition submodule is used to periodically sample each target event in a preset target event set to acquire operation information of each target event, and the target event set includes at least one target event.

[0160] In some embodiments, the detection module 402 may specifically include:

[0161] a detection submodule, configured to detect the big data application object according to the operation information of at least one target event, and identify at least one target redundant behavior in the big data application object, wherein the target redundant behavior corresponds to a target redundant type, and the target redundant behavior includes one of a dead write behavior, a silent write behavior, and a redundant read behavior;

[0162] The first construction submodule is used to construct redundant instances corresponding to each target redundant behavior according to the calling path and redundant ratio of each target redundant behavior, wherein the calling path is constructed by the source code location information of the target redundant behavior and the corresponding precise instruction address, and the redundant ratio is the ratio of the number of redundant bytes corresponding to the calling path of the target redundant behavior to the total number of accessed bytes;

[0163] A first generating submodule, configured to generate a redundancy subreport of each target redundancy type according to at least one redundancy instance of each target redundancy type;

[0164] The second generating submodule is used to generate a redundant report of the big data application object according to the redundant subreport of the dead write type, the redundant subreport of the silent write type and the redundant subreport of the redundant read type.

[0165] In some embodiments, the first construction submodule may specifically include:

[0166] An acquisition unit, used for acquiring source code location information of each target redundant behavior;

[0167] A first construction unit is used to construct a call path of each target redundant behavior according to source code location information of each target redundant behavior and a corresponding precise instruction address;

[0168] A calculation unit, used to calculate the redundancy ratio of each target redundant behavior according to the number of redundant bytes and the total number of access bytes corresponding to the call path of each target redundant behavior;

[0169] The second construction unit is used to construct redundant instances corresponding to each target redundant behavior according to the calling path and redundant ratio of each target redundant behavior.

[0170] In some embodiments, the above-mentioned calculation unit may specifically include:

[0171] A first calculation subunit is used to calculate the redundancy ratio of each hard-write behavior according to the number of redundant bytes corresponding to the call path of each hard-write behavior and the total number of written bytes when the target redundancy type is a hard-write type and the target redundant behavior is a hard-write behavior;

[0172] A second calculation subunit is used to calculate the redundancy ratio of each silent write behavior according to the number of redundant bytes corresponding to the call path of each silent write behavior and the total number of written bytes when the target redundancy type is a silent write type and the target redundancy behavior is a silent write behavior;

[0173] The third calculation subunit is used to calculate the redundancy ratio of each redundant read behavior according to the number of redundant bytes corresponding to the calling path of each redundant read behavior and the total number of read bytes when the target redundancy type is a redundant read type and the target redundant behavior is a redundant read behavior.

[0174] In some embodiments, the processing module 403 may specifically include:

[0175] A classification submodule, used for classifying each calling function in the call tree of at least one redundant instance in the redundant report to obtain at least one node of each redundant instance, where different nodes correspond to different calling functions;

[0176] A merging submodule, used to merge at least one redundant instance according to the same node, and sum up the redundant ratio and value of each node according to the redundant ratio of the corresponding calling function of each node;

[0177] A generation submodule is used to obtain a redundant information graph of a big data application object based on at least one merged node, the redundancy ratio and value of each node, and the calling relationship between each calling function in a calling tree of at least one redundant instance.

[0178] In some embodiments, the display module 404 may specifically include:

[0179] A first determination submodule is used to determine, in at least one node of the redundant information graph, a node whose redundancy ratio and value is greater than a preset threshold as a target node;

[0180] A second determination submodule is used to determine a node whose call relationship with the target node is extended as an extended node;

[0181] A second construction submodule is used to construct a target path of a target node according to the extended node;

[0182] The marking display submodule is used to mark the target node and the target path in the redundant information graph and display the redundant information graph.

[0183] Figure 5 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application is shown.

[0184] The electronic device may include a processor 501 and a memory 502 storing computer program instructions.

[0185] Specifically, the processor 501 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0186] The memory 502 may include a large capacity memory for data or instructions. By way of example and not limitation, the memory 502 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive or a combination of two or more of these. In appropriate cases, the memory 502 may include a removable or non-removable (or fixed) medium. In appropriate cases, the memory 502 may be inside or outside the integrated gateway disaster recovery device. In a specific embodiment, the memory 502 is a non-volatile solid-state memory.

[0187] In certain embodiments, the memory 502 may include a read-only memory (ROM), a random access memory (RAM), a magnetic disk storage medium device, an optical storage medium device, a flash memory device, an electrical, optical or other physical / tangible memory storage device. Thus, in general, the memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., a memory device) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the method according to an aspect of the present disclosure.

[0188] The processor 501 implements any one of the data detection methods in the above embodiments by reading and executing computer program instructions stored in the memory 502 .

[0189] In one example, the electronic device may further include a communication interface 503 and a bus 510. Figure 5 As shown, the processor 501, the memory 502, and the communication interface 503 are connected via a bus 510 and communicate with each other.

[0190] The communication interface 503 is mainly used to implement communication between various modules, devices, units and / or equipment in the embodiments of the present application.

[0191] Bus 510 includes hardware, software or both, and the parts of electronic equipment are coupled to each other. For example, but not limitation, bus may include accelerated graphics port (AGP) or other graphics bus, enhanced industrial standard architecture (EISA) bus, front side bus (FSB), hypertransport (HT) interconnection, industrial standard architecture (ISA) bus, infinite bandwidth interconnection, low pin count (LPC) bus, memory bus, micro channel architecture (MCA) bus, peripheral component interconnection (PCI) bus, PCI-Express (PCI-X) bus, serial advanced technology attachment (SATA) bus, video electronics standard association local (VLB) bus or other suitable bus or two or more of these combinations. In appropriate cases, bus 510 may include one or more buses. Although the present application embodiment describes and shows a specific bus, the application considers any suitable bus or interconnection.

[0192] The electronic device can execute the data detection method in the embodiment of the present application, thereby realizing the combination Figure 1 and Figure 4 Described data detection method and device.

[0193] In addition, in combination with the data detection method in the above embodiments, the present application embodiment can provide a computer-readable storage medium for implementation. The computer-readable storage medium stores computer program instructions; when the computer program instructions are executed by a processor, any one of the data detection methods in the above embodiments is implemented.

[0194] In combination with the data detection method in the above embodiments, an embodiment of the present application may provide a computer program product. When the instructions in the computer program product are executed by a processor of an electronic device, the electronic device executes any one of the data detection methods above.

[0195] It should be clear that the present application is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present application is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present application.

[0196] The functional blocks shown in the above-described block diagram can be implemented as hardware, software, firmware or a combination thereof. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present application are programs or code segments that are used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link by a data signal carried in a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, optical fiber media, radio frequency (RF) links, etc. The code segment can be downloaded via a computer network such as the Internet, an intranet, etc.

[0197] It should also be noted that the exemplary embodiments mentioned in this application describe some methods or systems based on a series of steps or devices. However, this application is not limited to the order of the above steps, that is, the steps can be performed in the order mentioned in the embodiment, or in a different order from the embodiment, or several steps can be performed simultaneously.

[0198] Aspects of the present disclosure are described above with reference to the flowchart and / or block diagram of the method, device (system) and computer program product according to the embodiment of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device to produce a machine so that these instructions executed by the processor of the computer or other programmable data processing device enable the implementation of the function / action specified in one or more boxes of the flowchart and / or block diagram. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field programmable logic circuit. It can also be understood that each box in the block diagram and / or flowchart and the combination of boxes in the block diagram and / or flowchart can also be implemented by dedicated hardware that performs a specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions.

[0199] The above is only a specific implementation of the present application. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working processes of the systems, modules and units described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here. It should be understood that the protection scope of the present application is not limited to this. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in this application, and these modifications or replacements should be included in the protection scope of this application.

Claims

1. A data detection method, characterized in that: include: When the big data application object is running, obtaining running information of at least one target event in the big data application object, the running information including precise instruction address, memory access type, memory access address, number of bytes and byte value; According to the operation information of the at least one target event, the big data application object is detected to generate a redundancy report of the big data application object, wherein the redundancy report includes at least one redundancy instance, and the redundancy instance includes a call path and a redundancy ratio; Processing the redundancy report to obtain a redundancy information graph of the big data application object, the redundancy information graph including at least one node and a redundancy ratio and value of each node, the at least one node being obtained by classifying each calling function in a call tree of the at least one redundant instance, different nodes corresponding to different calling functions, and the redundancy ratio and value being the sum of the redundancy ratios in the calling functions corresponding to the nodes; The target node and the target path in the redundant information graph are displayed, wherein the target node is a node in the at least one node of the redundant information graph whose redundancy ratio and value is greater than a preset threshold, and the target path is a path associated with the target node.

2. The method according to claim 1, characterized in that When the big data application object is running, obtaining the running information of at least one target event in the big data application object includes: Configure the environment parameters of the big data application object according to the target redundancy type, wherein the target redundancy type includes any one of a hard-write type, a silent write type, and a redundant read type; Based on the environmental parameters, running the big data application object; Each target event in a preset target event set is sampled periodically to obtain operation information of each target event, wherein the target event set includes at least one target event.

3. The method according to claim 2, characterized in that The detecting the big data application object according to the operation information of the at least one target event and generating a redundancy report of the big data application object includes: Detecting the big data application object according to the running information of the at least one target event, identifying at least one target redundant behavior in the big data application object, the target redundant behavior corresponding to the target redundant type, the target redundant behavior including one of a dead write behavior, a silent write behavior, and a redundant read behavior; According to the calling path and redundancy ratio of each target redundant behavior, a redundant instance corresponding to each target redundant behavior is constructed, wherein the calling path is constructed by the source code location information of the target redundant behavior and the corresponding precise instruction address, and the redundancy ratio is the ratio of the number of redundant bytes corresponding to the calling path of the target redundant behavior to the total number of access bytes; generating a redundancy sub-report of each target redundancy type according to at least one redundancy instance of each target redundancy type; A redundant report of the big data application object is generated according to the redundant sub-report of the dead-write type, the redundant sub-report of the silent-write type, and the redundant sub-report of the redundant-read type.

4. The method according to claim 3, characterized in that The constructing, according to the calling path and the redundancy ratio of each of the target redundant behaviors, a redundant instance corresponding to each of the target redundant behaviors comprises: Obtaining source code location information of each of the target redundant behaviors; Constructing a call path for each of the target redundant behaviors according to the source code location information of each of the target redundant behaviors and the corresponding precise instruction address; Calculating the redundancy ratio of each target redundant behavior according to the number of redundant bytes and the total number of access bytes corresponding to the call path of each target redundant behavior; According to the calling path and the redundancy ratio of each of the target redundant behaviors, a redundant instance corresponding to each of the target redundant behaviors is constructed.

5. The method according to claim 4, characterized in that The calculating the redundancy ratio of each target redundant behavior according to the number of redundant bytes and the total number of access bytes corresponding to the calling path of each target redundant behavior includes: When the target redundancy type is the hard-write type and the target redundancy behavior is the hard-write behavior, calculating the redundancy ratio of each hard-write behavior according to the number of redundant bytes corresponding to the call path of each hard-write behavior and the total number of written bytes; When the target redundancy type is the silent write type and the target redundancy behavior is the silent write behavior, the redundancy ratio of each silent write behavior is calculated according to the number of redundant bytes corresponding to the call path of each silent write behavior and the total number of written bytes; When the target redundancy type is the redundant read type and the target redundant behavior is the redundant read behavior, the redundancy ratio of each redundant read behavior is calculated according to the number of redundant bytes corresponding to the calling path of each redundant read behavior and the total number of read bytes.

6. The method according to claim 1, characterized in that The step of processing the redundant report to obtain a redundant information graph of the big data application object includes: Classify each calling function in the calling tree of at least one redundant instance in the redundant report to obtain at least one node of each redundant instance, where different nodes correspond to different calling functions; Merging the at least one redundant instance according to the same node, and summing up the redundant ratios of the calling functions corresponding to the nodes to obtain the redundant ratios and values ​​of the nodes; A redundant information graph of the big data application object is obtained based on the merged at least one node, the redundancy ratio and value of each node, and the calling relationship between each calling function in the calling tree of the at least one redundant instance.

7. The method according to claim 1, characterized in that The display of the target node and the target path in the redundant information graph includes: Among the at least one node in the redundant information graph, determining a node whose redundancy ratio and value is greater than a preset threshold as a target node; Determine a node whose call relationship with the target node is extended as an extended node; Constructing a target path of the target node according to the extended node; The target node and the target path are marked in the redundant information graph, and the redundant information graph is displayed.

8. A data detection device, characterized in that: The device comprises: an acquisition module, used for acquiring, when the big data application object is running, the running information of at least one target event in the big data application object, the running information including the precise instruction address, the memory access type, the memory access address, the number of bytes and the byte value; a detection module, configured to detect the big data application object according to the operation information of the at least one target event, and generate a redundancy report of the big data application object, wherein the redundancy report includes at least one redundancy instance, and the redundancy instance includes a call path and a redundancy ratio; A processing module, configured to process the redundancy report to obtain a redundancy information graph of the big data application object, wherein the redundancy information graph includes at least one node and a redundancy ratio and value of each node, wherein the at least one node is obtained by classifying each calling function in a call tree of the at least one redundant instance, wherein different nodes correspond to different calling functions, and the redundancy ratio and value are the sum of the redundancy ratios in the calling functions corresponding to the nodes; A display module is used to display the target node and target path in the redundant information graph, wherein the target node is a node in the at least one node of the redundant information graph whose redundancy ratio and value are greater than a preset threshold, and the target path is a path associated with the target node.

9. An electronic device, characterized in that: The device comprises: a processor and a memory storing computer program instructions; when the processor executes the computer program instructions, the data detection method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer program instructions, and when the computer program instructions are executed by a processor, the data detection method according to any one of claims 1 to 7 is implemented.

11. A computer program product, characterized in that When the instructions in the computer program product are executed by a processor of an electronic device, the electronic device executes the data detection method as described in any one of claims 1 to 7.

Images