Data capture architecture fault diagnosis method, device, electronic device and storage medium

Abstract

The method, device, electronic device and storage medium for diagnosing faults in a data capture architecture provided by the embodiments of the present application include: obtaining a data capture architecture to be diagnosed; wherein the data capture architecture is an architecture for capturing data composed of multiple components; regularly collecting index data of each component, and drawing data change curves corresponding to each component based on the index data; determining whether the data capture architecture has a fault based on the data change curves of each component; and when it is determined that the data capture architecture has a fault, determining the faulty component in the data capture architecture based on the data change curves of each component. In the above technical solution, it is possible to automatically determine which specific faulty component in the data capture architecture is, and the fault diagnosis process of the data capture architecture does not require manual intervention. Compared with the existing method of manually checking the data capture architecture, this method of fault diagnosis greatly improves efficiency and effectively reduces the phenomenon of missed detection and false detection.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of computer technology, and in particular to a method, device, electronic device and storage medium for diagnosing a data capture architecture fault. Background Art

[0002] The data capture architecture can efficiently and stably capture data from various data sources to meet the needs of subsequent data analysis, mining and application.

[0003] Typically, a data capture architecture includes multiple core components for implementing data capture. Currently, only users can determine whether an abnormality has occurred in the data capture architecture after receiving the data, but it is impossible to determine in detail which component has a problem. In order to solve the above problem, it is currently necessary to manually check each component of the data capture architecture. Since a large amount of mechanical work needs to be done manually, the existing fault diagnosis method is inefficient, and suffers from missed detections and false detections. Summary of the invention

[0004] In view of this, in order to effectively alleviate the technical problems of inefficiency, missed detection and false detection caused by manual inspection of various components of the data capture architecture, an embodiment of the present invention provides a method, device, electronic device and storage medium for data capture architecture fault diagnosis.

[0005] In a first aspect, an embodiment of the present invention provides a method for diagnosing a data capture architecture fault, the method comprising: Obtain a data capture architecture for a fault to be diagnosed; wherein the data capture architecture is an architecture composed of multiple components for capturing data; Regularly collect the index data of each component, and draw the data change curve corresponding to each component based on the index data; Determine whether the data capture architecture has a fault based on the data change curves of each component; When it is determined that a fault exists in the data capture architecture, the faulty component in the data capture architecture is determined based on the data change curves of each component.

[0006] In a possible implementation, the data capture architecture is sequentially composed of a first flume component, a kafka component, a second flume component, and an hdfs component; Regularly collect indicator data of each component, including: Regularly collect the first inbound network card traffic and the first outbound network card traffic of the first flume component, the second inbound network card traffic and the second outbound network card traffic of the kafka component, the third inbound network card traffic and the third outbound network card traffic of the second flume component, and the inventory increase of the hdfs component.

[0007] In a possible implementation, drawing a data change curve corresponding to each component based on the indicator data includes: Draw a first data change curve of the first flume component based on the first network inbound card flow of the first flume component; Draw a second data change curve of the first flume component based on the first outbound network card traffic of the first flume component; Draw a third data change curve of the Kafka component based on the second inbound network card traffic of the Kafka component; Draw a fourth data change curve of the Kafka component based on the second outbound network card traffic of the Kafka component; Draw a fifth data change curve of the second flume component based on the third inbound network card traffic of the second flume component; Draw a sixth data change curve of the second flume component based on the third outgoing network card traffic of the second flume component; The seventh data change curve of the hdfs component is drawn based on the inventory increase of the hdfs component.

[0008] In a possible implementation, determining whether a data capture architecture has a fault based on the data change curves of each component includes: Obtain the slope of the data change curve of each component; Determine whether the difference in slope of each curve is within a preset range; When it is determined that the difference between the slopes of the various curves is within a preset range, it is determined that the data capture architecture is normal; When it is determined that at least one of the differences in the slopes of the curves is not within a preset range, it is determined that a fault exists in the data capture architecture.

[0009] In a possible implementation, determining a faulty component in a data capture architecture according to a data change curve of each component includes: When it is detected that the slope of the curve in the fourth data change curve is smaller than the slope of the curve in the third data change curve, it is determined that the faulty component in the data capture architecture is the Kafka component; When it is detected that the slope of the curve in the seventh data change curve is greater than the slope of the curve in the sixth data change curve, it is determined that the faulty component in the data capture architecture is the hdfs component; When it is monitored that the data change curves of each component are all preset curves, it is determined based on the preset dial test method that the faulty component in the data capture architecture is the first flume component and / or the kafka component.

[0010] In a possible implementation, determining that the faulty component in the data capture architecture is the first flume component and / or the kafka component based on a preset dial test method includes: Send a POST request to the server where the first Flume component is located; Receive the response status code for the POST request; Determine whether the response status code is a preset code; When it is determined that the response status code is not a preset code, determining that the faulty component in the data capture architecture is the first flume component; When it is determined that the response status code is a preset code, determining that the first flume component is normal; and, Push test data to the specified topic in the Kafka component through push instructions; Consume the test data of the specified topic in the Kafka component through the consumption instruction; Determine whether the pushed test data is consistent with the consumed test data; When it is determined that the pushed test data and the consumed test data are inconsistent, it is determined that the faulty component in the data capture architecture is the Kafka component; When it is determined that the pushed test data is consistent with the consumed test data, it is determined that the Kafka component is normal.

[0011] In one possible implementation, the method further includes: When it is determined that the faulty component in the data capture architecture is the Kafka component, query instructions are used to obtain the lag of a specific consumer group in the Kafka component on each partition. The lag of each partition is used to obtain the total accumulated data volume corresponding to the Kafka component. In a second aspect, an embodiment of the present invention provides a device for diagnosing a data capture architecture fault, the device comprising: An acquisition module is used to acquire a data capture architecture for fault diagnosis; wherein the data capture architecture is an architecture for capturing data composed of multiple components; The collection and drawing module is used to regularly collect the index data of each component and draw the data change curve corresponding to each component based on the index data; A determination module, used to determine whether the data capture architecture has a fault based on the data change curves of each component; The fault diagnosis module is used to determine the faulty component in the data capture architecture based on the data change curves of each component when it is determined that the data capture architecture has a fault.

[0012] In a third aspect, an embodiment of the present invention provides an electronic device, which includes: a processor and a memory, the processor is used to execute a data capture architecture fault diagnosis program stored in the memory to implement the above-mentioned data capture architecture fault diagnosis method.

[0013] In a fourth aspect, an embodiment of the present invention provides a storage medium, wherein the storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the above-mentioned method of data capture architecture fault diagnosis.

[0014] The method, device, electronic device and storage medium for diagnosing faults in a data capture architecture provided by an embodiment of the present invention include: obtaining a data capture architecture to be diagnosed; wherein the data capture architecture is an architecture for capturing data composed of multiple components; regularly collecting index data of each component, and drawing data change curves corresponding to each component based on the index data; determining whether the data capture architecture has a fault based on the data change curves of each component; and determining the faulty component in the data capture architecture based on the data change curves of each component when it is determined that the data capture architecture has a fault. In the above technical scheme, it is possible to automatically determine whether the data capture architecture has a fault based on the data change curve drawn by the index data of each component in the data capture architecture, and when it is determined that the data capture architecture has a fault, it is further possible to automatically determine which specific faulty component in the data capture architecture is based on the data change curves of each component. The fault diagnosis process of the data capture architecture does not require manual intervention. Compared with the existing method of manually checking the data capture architecture, this fault diagnosis method greatly improves efficiency and effectively reduces the phenomenon of missed detection and false detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 A flowchart of an embodiment of a method for diagnosing a data capture architecture fault provided by an embodiment of the present invention; Figure 2 A schematic diagram of a data capture architecture provided by an embodiment of the present invention; Figure 3 A schematic diagram of an indicator data collection provided by an embodiment of the present invention; Figure 4 A schematic diagram of indicator data provided by an embodiment of the present invention; Figure 5 A schematic diagram of a data change curve provided by an embodiment of the present invention; Figure 6 A flowchart of another method for diagnosing a data capture architecture fault provided by an embodiment of the present invention; Figure 7 A flowchart of another method for diagnosing a data capture architecture fault provided by an embodiment of the present invention; Figure 8 A block diagram of an embodiment of a device for diagnosing a data capture architecture fault provided by an embodiment of the present invention; Fig. 9 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0016] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0017] To facilitate understanding of the embodiments of the present invention, specific embodiments will be further explained below in conjunction with the accompanying drawings. The embodiments do not constitute a limitation on the embodiments of the present invention.

[0018] The present invention provides a method for diagnosing a data capture architecture fault. Figure 1 , Figure 1 A flow chart of an embodiment of a method for diagnosing faults in a data capture architecture provided in an embodiment of the present invention. Figure 1 The process shown may include the following steps: Step 101, obtaining a data capture architecture for a fault to be diagnosed; The acquired data capture architecture is a data capture architecture that is currently performing data capture and needs to be fault diagnosed. In order to be able to check in real time whether there is a fault in the data capture architecture, the method provided in this embodiment can be used to perform real-time automatic fault diagnosis on the acquired data capture frame.

[0019] In actual applications, in order to accurately and completely capture data from the data source, the above-mentioned data capture architecture needs to include multiple components with functions such as data collection, data transmission, and data storage. Therefore, the data capture architecture is an architecture for capturing data composed of multiple components.

[0020] Step 102, regularly collecting index data of each component, and drawing a data change curve corresponding to each component based on the index data; The above indicator data are data that can be used to diagnose faults in the data capture architecture and components. It can be understood that different components may collect different corresponding indicator data due to their different functions in the data capture process. This will be explained in detail below and will not be repeated here.

[0021] In specific implementation, the indicator data of each component needs to be collected regularly and simultaneously in order to draw the data change curve of each component over time. The regular time interval can be 1 second, 2 seconds or 3 seconds, which can be set according to actual needs and is not limited here.

[0022] Since the curve graph can intuitively show the trend of data changes over time, this trend analysis is crucial for fault diagnosis. Through the fluctuation of the curve, slope change and other characteristics, abnormal points or trends in the data can be quickly discovered, so as to timely determine whether the data capture architecture is faulty and the faulty components in the data capture architecture. Therefore, in this embodiment, the index data of each component collected is plotted into the data change curve corresponding to each component, so as to facilitate the subsequent data capture architecture and timely and accurate determination of component failures.

[0023] Step 103, determining whether the data capture architecture has a fault based on the data change curves of each component; Since the data change curve of each component is drawn by the indicator data of each component, and the indicator data of each component is data that can realize the fault diagnosis of the data capture architecture, therefore, in this embodiment, it is possible to automatically and quickly determine whether the data capture architecture has a fault based on the data change curve of each component. When it is determined that the data capture architecture has a fault through the data change curve of each component, in order to further determine which specific faulty component in the data capture architecture is, step 104 needs to be executed. When it is determined that the data capture architecture is normal through the data change curve of each component, there is no need to execute step 104, and data capture and step 102 can be continued.

[0024] Step 104 , when it is determined that there is a fault in the data capture architecture, the faulty component in the data capture architecture is determined according to the data change curves of each component.

[0025] In this embodiment, it is deemed that the data source connected to the data capture architecture can send data normally. Then, when there is a fault in the data capture architecture, there must be a fault in a component in the data capture architecture. From the above description, it can be seen that since the data change curve of each component is drawn by the indicator data of each component, and the indicator data of each component is data that can realize component fault diagnosis, therefore, in this embodiment, it is possible to automatically and quickly determine which specific component or components in the data capture architecture have a fault based on the data change curve of each component, and automatic diagnosis of the data capture architecture and component faults in the data capture architecture can be realized.

[0026] The method for diagnosing faults in a data capture architecture provided by an embodiment of the present invention comprises: obtaining a data capture architecture to be diagnosed; wherein the data capture architecture is an architecture for capturing data composed of multiple components; regularly collecting index data of each component, and drawing data change curves corresponding to each component based on the index data; determining whether the data capture architecture has a fault based on the data change curves of each component; and determining the faulty component in the data capture architecture based on the data change curves of each component when it is determined that the data capture architecture has a fault. In the above technical scheme, it is possible to automatically determine whether the data capture architecture has a fault based on the data change curve drawn by the index data of each component in the data capture architecture, and when it is determined that the data capture architecture has a fault, it is further possible to automatically determine which specific faulty component in the data capture architecture is based on the data change curves of each component. The fault diagnosis process of the data capture architecture does not require manual intervention. Compared with the existing method of manually checking the data capture architecture, this fault diagnosis method greatly improves efficiency and effectively reduces the phenomenon of missed detection and false detection. In one implementation, the data capture architecture is sequentially composed of a first flume component, a kafka component, a second flume component, and an hdfs (Hadoop Distributed File System) component.

[0027] For ease of understanding, Figure 2 As shown, the first flume component 201 in the data capture architecture 200 is connected to the data source 202. The first flume component 201 first collects data from the data source through its Source component. The data source can be a log file, network data, sensor data, etc., which is not limited here. The Source component encapsulates the data into Event (including Header and Body) and passes it to the Channel component. Then the Channel component passes the data to the Sink component. The Sink component sends the data to the kafka component 203. As a distributed, high-throughput message system, the kafka component 203 can efficiently handle the transmission of large amounts of data and ensure the reliability and sequentiality of the data. The second flume component 204, as a consumer of the kafka component 203, pulls data from the kafka component 203, and uses the Source component to pull data from the kafka component 203. After that, the data is encapsulated into Event again and passed to its Channel component. Then, the Channel component passes the data to the Sink component, and the Sink component writes the data to the hdfs component 205 for storage. ‌‌‌‌‌‌

[0028] Based on the above description, as an optional implementation method, the process of regularly collecting indicator data of each component in the above step 102 is: regularly collecting the first network inlet traffic and the first network outlet traffic of the first flume component, the second network inlet traffic and the second network outlet traffic of the kafka component, the third network inlet traffic and the third network outlet traffic of the second flume component, and the inventory increase of the hdfs component.

[0029] For ease of understanding, Figure 3 As shown, the first network inlet traffic is the amount of data collected by the Source component in the first flume component, the first network outlet traffic is the amount of data transmitted by the Sink component in the first flume component, the second network inlet traffic is the amount of data received by the kafka component from the first flume component, the second network outlet traffic is the amount of data sent by the kafka component, the third network inlet traffic is the amount of data received by the Source component in the second flume component, the third network outlet traffic is the amount of data transmitted by the Sink component in the second flume component, and the inventory increase is the amount of data currently stored in the hdfs component.

[0030] Among them, the collection of indicator data corresponding to the first flume component, the kafka component and the second flume component can be done by logging in to the servers where the listed components are independently deployed and using the dsata command to filter the recv and send fields, which correspond to the incoming network card traffic and the outgoing network card traffic, respectively. Figure 4 As shown, the recv and send fields respectively show the second inbound network card traffic and the second outbound network card traffic of the Kafka component collected regularly.

[0031] The collection of indicator data corresponding to the hdfs component can be done by logging in to the server where the hdfs component is deployed and using the command hdfs dfs du-sh to obtain the storage capacity of the underlying data of the hdfs component, and by comparing the storage capacity of the underlying data obtained last time, the indicator data of the regularly collected inventory increase of the hdfs component at the current moment can be determined. Specifically, the inventory increase of the hdfs component can also be collected by combining the use of ls and stat commands, using hdfs monitoring tools, using hdfs's dfsadmin-report command, etc., which will not be described in detail here.

[0032] In actual application, since there are 7 indicator data collected for the first flume component, the kafka component, the second flume component and the hdfs component, there are 7 corresponding data change curves drawn. In one embodiment, the data change curves corresponding to each component are drawn based on the indicator data, including: drawing a first data change curve of the first flume component based on the first inbound network card flow of the first flume component; drawing a second data change curve of the first flume component based on the first outbound network card flow of the first flume component; drawing a third data change curve of the kafka component based on the second inbound network card flow of the kafka component; drawing a fourth data change curve of the kafka component based on the second outbound network card flow of the kafka component; drawing a fifth data change curve of the second flume component based on the third inbound network card flow of the second flume component; drawing a sixth data change curve of the second flume component based on the third outbound network card flow of the second flume component; and drawing a seventh data change curve of the hdfs component based on the inventory increase of the hdfs component.

[0033] For ease of understanding, Figure 5 As shown, Figure 5 The data change curves displayed from top to bottom are the first data change curve, the second data change curve, the third data change curve, the fourth data change curve, the fifth data change curve, the sixth data change curve and the seventh data change curve. In actual application, since the data collected from the data source needs to be compressed when passing through the first flume component, the kafka component, the second flume component and the hdfs component, the data volume in the first flume component is the largest, and the data volume in the hdfs component is the smallest. Therefore, the data change curves corresponding to each component correspond to Figure 5 The situation shown.

[0034] like Figure 6 As shown, as an optional implementation, as in the above method, step 103 determines whether the data capture architecture has a fault based on the data change curves of each component, including the following steps: Step 601, obtaining the slope of the data change curve of each component; In this embodiment, whether the data capture architecture is abnormal can be determined by monitoring the curve change trends of the data change curves corresponding to each component. If the curve change trends of the data change curves corresponding to each component are consistent, it usually means that the data is correctly and synchronously processed and transmitted between the components, thereby determining that the data capture architecture is fault-free; on the contrary, if the curve change trends are inconsistent, it may indicate that there is a delay, loss or error in the data processing or transmission process, and therefore there is a fault in the data capture architecture. The curve change trend of the data change curve corresponding to each component is mainly determined by the curve slope. Therefore, it is necessary to obtain the curve slope of the data change curve of each component.

[0035] Step 602, determining whether the difference between the slopes of the various curves is within a preset range; Step 603, when it is determined that the difference between the slopes of the various curves is within a preset range, it is determined that the data capture architecture is normal; Step 604: When it is determined that at least one difference value among the differences in the slopes of the curves is not within a preset range, it is determined that a fault exists in the data capture architecture. In the data capture architecture, each component in the data capture architecture is responsible for processing different parts or stages of the data. If the data capture architecture works normally, the data processed by each component should remain synchronized, that is, their data change curves should show a consistent trend, that is, the difference in the slope of the curve should remain within the preset range. If there is at least one difference in the slope of the curve that exceeds the preset range, it indicates that the data has become asynchronous or inconsistent during processing or transmission, that is, there is a fault in the data capture architecture.

[0036] The difference in the slope of the curve exceeding the preset range is one of the important bases for determining that there is a fault in the data capture architecture. When this situation is found, the faulty component in the data capture architecture should be identified as soon as possible and corresponding measures should be taken to repair it to ensure the stability and accuracy of the data capture architecture.

[0037] like Figure 7 As shown, as an optional implementation, as in the above method, step 104 of determining the faulty component in the data capture architecture according to the data change curve of each component includes the following steps: Step 701: when it is monitored that the slope of the curve in the fourth data change curve is smaller than the slope of the curve in the third data change curve, it is determined that the faulty component in the data capture architecture is the Kafka component; When the slope of the curve in the fourth data change curve is smaller than the slope of the curve in the third data change curve, it indicates that the second outbound network card traffic of the Kafka component is greater than the second inbound network card traffic of the Kafka component. It can be determined that data backlog has occurred in the Kafka component, that is, the faulty component is determined to be the Kafka component in the data capture architecture.

[0038] Generally, in order to take corresponding measures to solve the data backlog problem of the Kafka component according to the total amount of accumulated data in the Kafka component, it is necessary to first determine the total amount of accumulated data corresponding to the Kafka component. In actual application, the lag of a specific consumer group in the Kafka component on each partition can be obtained through query instructions; the lag of each partition is used to obtain the total amount of accumulated data corresponding to the Kafka component.

[0039] The above query command is kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group<consumer_group_id> , used to query and obtain<consumer_group_id> The lag (LAG) of this particular consumer group on each partition, and then add up the lag on each partition to get the total accumulated data corresponding to the Kafka component.

[0040] After obtaining the total accumulated data volume corresponding to the Kafka component, if it is determined through the total accumulated data volume that there is a small amount of data backlog: you can improve the throughput and reliability of the Kafka component by optimizing its configuration. For example, adjust the num.partitions parameter to increase the number of partitions of the Kafka component to improve parallel processing capabilities; or adjust the message.max.bytes parameter to increase the maximum number of bytes of the message to accommodate larger messages.

[0041] If the total amount of accumulated data determines that there is a "medium data backlog": Consider increasing the number of consumers and increasing the speed of data consumption by adding consumer instances to the consumer group, which can directly reduce data accumulation.

[0042] If the producer sends data too quickly, you can adjust the producer's parameters, such as batch.size and linger.ms, to control the amount of data sent, thereby slowing down the production rate so that the consumer can keep up.

[0043] If the total amount of accumulated data indicates that there is a large data backlog: you need to consider solutions more comprehensively, including increasing the number of brokers or hardware resources of the Kafka component to improve the overall throughput.

[0044] At the same time, you can increase the number of topic partitions to further improve parallel processing capabilities, and consider further optimization of the consumer side, such as adopting more efficient data processing algorithms or increasing processing threads. In extreme cases, you may need to consider expanding or refactoring the Kafka component to better cope with large-scale data backlogs.

[0045] Step 702: when it is detected that the slope of the curve in the seventh data change curve is greater than the slope of the curve in the sixth data change curve, it is determined that the faulty component in the data capture architecture is the hdfs component; When the slope of the seventh data change curve is greater than the slope of the sixth data change curve, it means that the amount of data transmitted by the second Flume component is much greater than the storage capacity of the HDFS component. Then there may be problems with the health or network connection of the HDFS component. Therefore, it can be determined that the faulty component in the current data capture architecture is the HDFS component.

[0046] For HDFS components with health problems, the failure can be resolved by replacing new HDFS components or repairing HDFS components. For HDFS components with network connection problems, the failure can be resolved through strategies such as data localization, data replication, and data block size adjustment to ensure the stability and accuracy of the data crawling architecture.

[0047] Step 703: When it is monitored that the data change curves of each component are all preset curves, determine based on the preset dial test method that the faulty component in the data capture architecture is the first flume component and / or the kafka component.

[0048] The preset curve is a curve that characterizes problems in the data capture architecture. For example, if the preset curve is a zeroing curve, then the data change curves of each component are all zeroing curves, that is, the data capture architecture performs data capture. In this case, it is determined that the first flume component and / or the kafka component in the data capture architecture are faulty components. In actual application, the preset dialing test method can be used to specifically determine whether the first flume component is faulty, the kafka component is faulty, or both the first flume component and the kafka component are faulty.

[0049] Specifically, determining that the faulty component in the data capture architecture is the first flume component and / or the kafka component based on the preset dial test method can be achieved by steps A1 to A10: Step A1, sending a POST request to the server where the first flume component is located; Step A2, receiving a response status code for the POST request; The response status code can be used to indicate the processing result of the first flume component on the POST request.

[0050] Step A3, determining whether the response status code is a preset code; The preset code is a response status code indicating that the processing result of the first flume component for the POST request is normal. In this embodiment, the preset code can be set to 200. The specific code can also be set according to actual needs and is not limited here.

[0051] Step A4: when it is determined that the response status code is not a preset code, determining that the faulty component in the data capture architecture is the first flume component; When it is determined that the response status code is not the preset code, it indicates that there is an abnormality in the processing result of the first flume component on the POST request, that is, there is a fault in the first flume component. Therefore, the first flume component is determined as a faulty component in the data capture architecture.

[0052] Step A5, when it is determined that the response status code is a preset code, determining that the first flume component is normal; and, When it is determined that the response status code is the preset code, it indicates that there is no abnormality in the processing result of the first flume component on the POST request, that is, the first flume component is not a faulty component in the data capture architecture.

[0053] The command corresponding to the preset dial test method for the first flume component is: curl -v-XPOST-iH"Topic:testpasca"-H"Format:csv"-H"Row-Split:\\n"-H"Field-Split:,"–H"User:user00"–H"Password:xxxx"--data-binary @ / home / shuju.csv http: / / xxxx:10080.

[0054] Among them, curl: is a command line tool used to send and receive data and supports multiple protocols, including HTTP and HTTPS.

[0055] -v: Enable verbose mode to display detailed information of requests and responses. -XPOST: Specifies the request method as POST.

[0056] -i: Include HTTP response headers in the output.

[0057] -H: Add HTTP header fields to the request. In this command, multiple custom header fields are added, such as Topic, Format, Row-Split, Field-Split, User, and Password.

[0058] --data-binary: Send data in binary form, here is the content of the file / home / shuju.csv.

[0059] http: / / xxxx:10080: Target URL and port, that is, the service address to which the request is sent.

[0060] If the response status code returned by executing the command corresponding to the above preset dial test mode is 200, it means that the first flume is running normally. If the response status code is not 200, it means that the first flume has a fault.

[0061] Step A6: Push test data to the specified topic in the Kafka component through a push instruction; The preset dial test method for the Kafka component can be implemented through push instructions and consumption instructions.

[0062] The command corresponding to the push instruction is: kafka-console-producer--broker-listX.XXX:9092--topic test1. By executing the command corresponding to the push instruction, test data can be pushed to the specified topic --topic test1.

[0063] Step A7, consume the test data of the specified topic in the Kafka component through the consumption instruction; The command corresponding to the above consumption instruction is: kafka-console-consumer --bootstrap-server XXXX:9092--topic test1--new-consumer. By executing the command corresponding to the consumption instruction, you can consume test data from the specified topic --topic test1 in the kafka component.

[0064] Step A8, determining whether the pushed test data is consistent with the consumed test data; Step A9: when it is determined that the pushed test data and the consumed test data are inconsistent, it is determined that the faulty component in the data capture architecture is the Kafka component; If it is determined that the pushed test data and the consumed test data are inconsistent, it means that the test data pushed to the Kafka component cannot be consumed correctly, that is, the data cannot be successfully pushed from the producer to the consumer, and there is a fault in the Kafka component. Therefore, the Kafka component is determined to be a faulty component in the data capture architecture.

[0065] Step A10: When it is determined that the pushed test data is consistent with the consumed test data, it is determined that the Kafka component is normal.

[0066] If it is determined that the pushed test data is consistent with the consumed test data, it means that the test data pushed to the Kafka component can be consumed correctly, that is, the data can be successfully pushed from the producer to the consumer, and there is no fault in the Kafka component, that is, the Kafka component is not a faulty component in the data capture architecture.

[0067] In actual application, the preset dial test method for the first flume component and the preset dial test method for the kafka component can be performed simultaneously to speed up the efficiency of determining the faulty component. In specific implementation, they can also be executed sequentially, which is not limited here.

[0068] Through the method provided in this embodiment, it is possible to determine through a preset dialing test mode whether the first flume component is faulty, whether the kafka component is faulty, or whether both the first flume component and the kafka component are faulty. This dialing test function using the data change curve assisted by the background component can simply, efficiently, and intuitively determine the faulty components in the data capture architecture, providing strong support for resuming data capture as soon as possible.

[0069] See also Figure 8 , is a block diagram of an embodiment of a device for diagnosing a data capture architecture fault provided by an embodiment of the present invention. Figure 8 As shown, the device comprises: Acquisition module 801, used to acquire a data capture architecture for fault diagnosis; wherein the data capture architecture is an architecture for capturing data composed of multiple components; The collection and drawing module 802 is used to regularly collect the index data of each component and draw the data change curve corresponding to each component based on the index data; A determination module 803, used to determine whether the data capture architecture has a fault based on the data change curves of each component; The fault diagnosis module 804 is used to determine the faulty component in the data capture architecture according to the data change curves of each component when it is determined that the data capture architecture has a fault.

[0070] The device for diagnosing faults in a data capture architecture provided by an embodiment of the present invention comprises: obtaining a data capture architecture to be diagnosed; wherein the data capture architecture is an architecture for capturing data composed of multiple components; regularly collecting index data of each component, and drawing data change curves corresponding to each component based on the index data; determining whether the data capture architecture has a fault based on the data change curves of each component; and determining the faulty component in the data capture architecture according to the data change curves of each component when it is determined that the data capture architecture has a fault. In the above technical scheme, it is possible to automatically determine whether the data capture architecture has a fault based on the data change curve drawn by the index data of each component in the data capture architecture, and when it is determined that the data capture architecture has a fault, it is further possible to automatically determine which specific faulty component in the data capture architecture is based on the data change curves of each component. The fault diagnosis process of the data capture architecture does not require manual intervention. Compared with the existing method of manually checking the data capture architecture, this fault diagnosis method greatly improves efficiency and effectively reduces the phenomenon of missed detection and false detection. Fig. 9 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention, Fig. 9 The electronic device 500 shown includes: at least one processor 501, a memory 502, at least one network interface 504 and other user interfaces 503. The various components in the electronic device 500 are coupled together via a bus system 505. It is understood that the bus system 505 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 505 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, the bus system 505 is not shown in FIG. Fig. 9 Various buses are labeled as bus system 505 .

[0071] The user interface 503 may include a display, a keyboard, or a pointing device (eg, a mouse, a trackball, a touch pad, or a touch screen).

[0072] It can be understood that the memory 502 in the embodiment of the present invention can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus random access memory (DRRAM). The memory 502 described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0073] In some implementations, the memory 502 stores the following elements, executable units or data structures, or a subset thereof, or an extended set thereof: an operating system 5021 and an application program 5022 .

[0074] The operating system 5021 includes various system programs, such as a framework layer, a core library layer, a driver layer, etc., which are used to implement various basic services and process hardware-based tasks. The application 5022 includes various application programs, such as a media player (Media Player), a browser (Browser), etc., which are used to implement various application services. The program for implementing the method of the embodiment of the present invention can be included in the application 5022.

[0075] In the embodiment of the present invention, by calling the program or instruction stored in the memory 502, specifically, the program or instruction stored in the application 5022, the processor 501 is used to execute the method steps provided by each method embodiment.

[0076] The method disclosed in the above embodiment of the present invention can be applied to the processor 501, or implemented by the processor 501. The processor 501 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit in the processor 501 or the instruction in the form of software. The above processor 501 can be a general processor, a digital signal processor (Digital Signal Processor, DSP), an application specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field programmable gate array (Field Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present invention can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in the embodiment of the present invention can be directly embodied as a hardware decoding processor to execute, or the hardware and software units in the decoding processor can be executed. The software unit can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 502, and the processor 501 reads the information in the memory 502 and completes the steps of the above method in combination with its hardware.

[0077] It is understood that the embodiments described herein may be implemented in hardware, software, firmware, middleware, microcode, or a combination thereof. For hardware implementation, the processing unit may be implemented in one or more application specific integrated circuits (ASIC), digital signal processors (DSP), digital signal processing devices (DSPDevice, DSPD), programmable logic devices (PLD), field programmable gate arrays (FPGA), general purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions described in the present application, or a combination thereof.

[0078] For software implementation, the technology described herein can be implemented by a unit that performs the functions described herein. The software code can be stored in a memory and executed by a processor. The memory can be implemented in the processor or outside the processor.

[0079] The electronic device provided in this embodiment may be Fig. 9 The electronic device shown in FIG. 1 may perform the following steps: Figure 1 , 6 -7 All steps of the method for data capture architecture fault diagnosis, and then implement Figure 1 , 6 -7 shows the technical effect of the method for diagnosing the fault of the data capture architecture. For details, please refer to Figure 1 , 6 -7 Related descriptions are omitted here for the sake of brevity.

[0080] The embodiment of the present invention further provides a storage medium (computer-readable storage medium). The storage medium here stores one or more programs. The storage medium may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a read-only memory, a flash memory, a hard disk or a solid-state drive; the memory may also include a combination of the above-mentioned types of memory.

[0081] When one or more programs in the storage medium can be executed by one or more processors, the above-mentioned method of data capture architecture fault diagnosis can be implemented.

[0082] The processor is used to execute the data capture architecture fault diagnosis program stored in the memory to implement the steps of the data capture architecture fault diagnosis method.

[0083] The professionals should further realize that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to the function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0084] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0085] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for diagnosing a data capture architecture fault, characterized in that: The method comprises: Obtaining a data capture architecture for a fault to be diagnosed; wherein the data capture architecture is an architecture composed of multiple components for capturing data; Regularly collect indicator data of each of the components, and draw data change curves corresponding to each of the components based on the indicator data; Determining whether the data capture architecture has a fault based on the data change curves of each of the components; When it is determined that the data capture architecture has a fault, the faulty component in the data capture architecture is determined according to the data change curves of each of the components.

2. The method according to claim 1, characterized in that The data capture architecture is composed of a first flume component, a kafka component, a second flume component and an hdfs component in sequence; The periodic collection of indicator data of each component includes: Regularly collect the first network inlet traffic and the first network outlet traffic of the first flume component, the second network inlet traffic and the second network outlet traffic of the kafka component, the third network inlet traffic and the third network outlet traffic of the second flume component, and the inventory increase of the hdfs component.

3. The method according to claim 2, characterized in that Drawing the data change curve corresponding to each of the components based on the indicator data includes: Draw a first data change curve of the first flume component based on the first network inbound card traffic of the first flume component; Draw a second data change curve of the first flume component based on the first outbound network card traffic of the first flume component; Draw a third data change curve of the Kafka component based on the second network inlet card traffic of the Kafka component; Draw a fourth data change curve of the Kafka component based on the second outbound network card traffic of the Kafka component; Draw a fifth data change curve of the second flume component based on the third network card flow of the second flume component; Draw a sixth data change curve of the second flume component based on the third outgoing network card traffic of the second flume component; A seventh data change curve of the hdfs component is drawn based on the inventory increase of the hdfs component.

4. The method according to claim 1, characterized in that: The determining whether the data capture architecture has a fault based on the data change curves of each of the components includes: Obtaining the slope of the data change curve of each of the components; Determining whether the difference between the slopes of the curves is within a preset range; In the case where it is determined that the difference between the slopes of the curves is within the preset range, it is determined that the data capture architecture is normal; When it is determined that at least one of the differences in the slopes of the curves is not within the preset range, it is determined that a fault exists in the data capture architecture.

5. The method according to claim 3, characterized in that: The determining of the faulty component in the data capture architecture according to the data change curve of each component includes: When it is monitored that the slope of the curve in the fourth data change curve is less than the slope of the curve in the third data change curve, it is determined that the faulty component in the data capture architecture is the kafka component; When it is monitored that the slope of the curve in the seventh data change curve is greater than the slope of the curve in the sixth data change curve, it is determined that the faulty component in the data capture architecture is the hdfs component; When it is monitored that the data change curves of each of the components are preset curves, it is determined based on a preset dialing test method that the faulty component in the data capture architecture is the first flume component and / or the kafka component.

6. The method according to claim 5, characterized in that The determining, based on a preset dial test method, that the faulty component in the data capture architecture is the first flume component and / or the kafka component includes: Send a POST request to the server where the first flume component is located; Receiving a response status code for the POST request; Determining whether the response status code is a preset code; When it is determined that the response status code is not the preset code, determining that the faulty component in the data capture architecture is the first flume component; In the case where it is determined that the response status code is the preset code, determining that the first flume component is normal; and, Push test data to the specified topic in the kafka component through push instructions; Consume the test data of the specified topic in the kafka component through consumption instructions; Determine whether the pushed test data is consistent with the consumed test data; When it is determined that the pushed test data and the consumed test data are inconsistent, determining that the faulty component in the data capture architecture is the kafka component; When it is determined that the pushed test data is consistent with the consumed test data, it is determined that the Kafka component is normal.

7. The method according to claim 5, characterized in that The method further comprises: When it is determined that the faulty component in the data capture architecture is the Kafka component, obtaining the lag amount of a specific consumer group in the Kafka component on each partition through a query instruction; The total accumulated data volume corresponding to the Kafka component is obtained by taking the lag amount of each partition.

8. A device for diagnosing data capture architecture faults, characterized in that: The device comprises: An acquisition module, used to acquire a data capture architecture for a fault to be diagnosed; wherein the data capture architecture is an architecture for capturing data composed of multiple components; A collection and drawing module, used for regularly collecting the index data of each of the components, and drawing the data change curve corresponding to each of the components based on the index data; A determination module, used for determining whether the data capture architecture has a fault based on the data change curves of each of the components; The fault diagnosis module is used to determine the faulty component in the data capture architecture according to the data change curves of each of the components when it is determined that the data capture architecture has a fault.

9. An electronic device, characterized in that: include: A processor and a memory, wherein the processor is used to execute a data capture architecture fault diagnosis program stored in the memory to implement the data capture architecture fault diagnosis method described in any one of claims 1 to 7.

10. A storage medium, characterized in that: The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the method for diagnosing data capture architecture faults as described in any one of claims 1 to 7.

Images