Big data cluster detection method and device, electronic device and storage medium
By obtaining the detection files of the big data cluster and accessing and detecting the operating status of the big data cluster, the problem of the failure to promptly detect the big data cluster in the existing technology is solved, the efficiency of enterprise business processing is improved, and automation and accuracy are achieved. detection.
Patent Information
- Application Number
- CN202011122700.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-10-20
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2040-10-20
AI Technical Summary
The existing technology cannot detect whether the big data cluster is stopped in time, which will affect the business processing efficiency of the enterprise.
By obtaining the detection files of the big data cluster, including accessing information and business data processing execution statements, accessing the big data cluster and performing business data processing and detection. If the access cannot be made or the business data processing time exceeds the set time, issue an early warning notice to discover the cluster in a timely manner. Stop running.
实现了及时发现大数据集群停止运行的问题,提高了企业业务处理的效率,避免了人力检测带来的不准确性和浪费。
Smart Images

Figure CN112231280B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of electronic information, and in particular to a detection method and device for a big data cluster, an electronic device, and a storage medium. Background Art
[0002] With the development of big data technology, more and more big data clusters (such as GreenPlum and HADOOP) are used to process enterprise business. However, the current detection of big data clusters cannot detect the situation where the big data cluster stops running. Only when users cannot use the big data cluster to handle business can they find that the big data cluster has stopped running, which greatly affects the efficiency of enterprise business processing. Summary of the invention
[0003] The inventor has discovered through research that when a big data cluster stops running, it is often impossible to access the big data cluster, or after access, the big data cluster is unable to complete the detection task of normal business data processing. Based on this, the present application provides a big data cluster detection method and device, electronic device and storage medium, aiming to solve the problem of how to promptly detect when a big data cluster has stopped running.
[0004] A method for detecting a big data cluster, comprising:
[0005] Acquire a detection file of a big data cluster, the detection file comprising at least a first detection file and a second detection file; the first detection file comprises access information for accessing the big data cluster; the second detection file comprises an execution statement for performing business data processing based on the big data cluster;
[0006] Accessing the big data cluster according to the first detection file, and issuing a warning notification if the big data cluster cannot be accessed or cannot be accessed within a first period of time;
[0007] If the big data cluster is accessed within the first time period, the business data of the big data cluster is processed according to the second detection file; if the duration of the business data processing is greater than the second time period, or the business data processing cannot be performed, the early warning notification is issued.
[0008] The above method, optionally, issuing the warning notification includes:
[0009] In the case where the big data cluster cannot be accessed, the big data cluster cannot be accessed within the first time period, the duration of the business data processing of the big data cluster is greater than the second time period, or the big data cluster cannot perform the business data processing, generating preset detection result information for characterizing that the big data cluster has stopped running;
[0010] Outputting the detection result information to a detection log file;
[0011] The preset early warning tool is triggered to read the detection log file, and send the early warning information corresponding to the detection result information to the preset user communication signal to prompt the big data cluster to stop running.
[0012] In the above method, optionally, the step of obtaining the detection file of the big data cluster includes:
[0013] According to the detection time point specified in the preset detection configuration file, obtain the detection file of the big data cluster preset corresponding to the detection time point, wherein the detection configuration file includes the corresponding relationship between the detection time point and the big data cluster, and the detection time point corresponds to the big data cluster in that, at the detection time point, the big data cluster is detected.
[0014] The above method may optionally further include, if the duration of business data processing is less than the second duration, determining that the big data cluster is in a running state.
[0015] In the above method, optionally, the access information at least includes:
[0016] The IP address of the big data cluster, the identifier of the big data cluster, and the access password of the big data cluster;
[0017] The accessing the big data cluster includes: accessing a host node of the big data cluster based on the access information.
[0018] In the above method, optionally, the detection file also includes a third detection file, the third detection file includes an execution statement for detecting the operating status of each node in the big data cluster, and the third detection file is used to detect the operating status of all nodes in the big data cluster.
[0019] The above method may optionally further include, in response to an instruction of a user input to add a new detection file, obtaining a new detection file and an identifier carried by the new detection file, and determining, based on the identifier, the big data cluster corresponding to the new detection file, wherein the detection file corresponds to the big data cluster, and the detection file is used to detect the big data cluster.
[0020] A detection device for a big data cluster, comprising:
[0021] An acquisition unit is used to acquire a detection file of a big data cluster, wherein the detection file includes at least a first detection file and a second detection file; the first detection file includes access information for accessing the big data cluster; the second detection file includes an execution statement for performing business data processing based on the big data cluster;
[0022] an access unit, configured to access the big data cluster according to the first detection file, and issue a warning notification if the big data cluster cannot be accessed or cannot be accessed within a first period of time;
[0023] A processing unit is used to perform the business data processing on the big data cluster according to the second detection file if the big data cluster is accessed within the first time period; if the time length of the business data processing is greater than the second time length, or the business data processing cannot be performed, issue the early warning notification.
[0024] An electronic device includes: a processor and a memory, wherein the memory is used to store a program; and the processor is used to run the program to implement the above-mentioned method for detecting a big data cluster.
[0025] A storage medium, characterized in that instructions are stored in the storage medium, and when the storage medium is run on a computer, the computer executes the method for detecting a big data cluster as claimed in claim 1.
[0026] The method and device described in the present application obtain detection files of a big data cluster, and the detection files include at least a first detection file and a second detection file; the first detection file includes access information for accessing the big data cluster; the second detection file includes execution statements for performing business data processing based on the big data cluster; the big data cluster is accessed according to the first detection file, and if the big data cluster cannot be accessed or cannot be accessed within a first time period, it means that the big data cluster has stopped running, and an early warning notification is issued, so that the problem of the big data cluster stopping running can be discovered in time; if the big data cluster is accessed within the first time period, but business data is processed for the big data cluster according to the second detection file; if the time period for business data processing is greater than the second time period, or business data processing cannot be performed, it means that the big data cluster has stopped running, and an early warning notification is issued, so that the problem of the big data cluster stopping running can be discovered in time. In summary, the method provided by the present application can discover the problem of the big data cluster stopping running in time. BRIEF DESCRIPTION OF THE DRAWINGS
[0027] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0028] Figure 1 A flowchart of a data cluster detection method provided in an embodiment of the present application;
[0029] Figure 2 An architectural diagram of a big data cluster detection platform provided in an embodiment of the present application;
[0030] Figure 3 A schematic diagram of the structure of a data cluster detection device provided in an embodiment of the present application;
[0031] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0032] At present, more and more enterprises are applying big data clusters to process business data. However, it is currently impossible to detect when the big data cluster stops running. The big data cluster can only be found to have stopped running when users cannot use the big data cluster to handle their business. This greatly affects the efficiency of the enterprise's business processing.
[0033] The inventor has discovered through research that when a big data cluster stops running, it is often impossible to access the big data cluster, or after access, the big data cluster cannot complete the detection task of normal business data processing. Based on this, the present application provides a big data cluster detection method, the purpose of which is to determine whether the big data cluster is in a running state by accessing the big data cluster and performing business data processing detection tasks on the big data cluster, and to issue an early warning notification when it is determined that the big data cluster is in a stopped state, so that the anomaly of the big data cluster can be discovered in a timely manner.
[0034] It should be noted that, in the present application, the big data cluster is in an operating state, which means that the big data cluster has the ability to process business data normally, and the big data cluster is in a stopped state, which means that the business data processing process in the big data cluster is in a stopped state and cannot provide external data processing services.
[0035] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.
[0036] Figure 1 A method for detecting a big data cluster provided in an embodiment of the present application may include the following steps:
[0037] S101. Obtain a detection file of a big data cluster.
[0038] In this embodiment, the detection file includes at least a first detection file and a second detection file; the first detection file includes access information for accessing the big data cluster; the second detection file includes execution statements for business data processing based on the big data cluster.
[0039] The access information includes: the IP address of the big data cluster, the identifier of the big data cluster, and the access password of the big data cluster. The identifier of the big data cluster may be a cluster user name.
[0040] The specific implementation method of this step may be: according to the detection time point specified in the preset detection configuration file, the detection file of the big data cluster corresponding to the preset detection time point is obtained regularly.
[0041] The detection configuration file includes a corresponding relationship between a detection time point and a big data cluster. The detection time point corresponds to the big data cluster in that the big data cluster is detected at the detection time point.
[0042] S102: Access a big data cluster according to the first detection file.
[0043] Specifically, the host node of the big data cluster is accessed using the IP address of the big data cluster, the identifier of the big data cluster, and the access password of the big data cluster.
[0044] S103: Determine whether the big data cluster is accessed within the first time period, or whether the big data cluster cannot be accessed. If the big data cluster is accessed within the first time period, execute S104. If the big data cluster cannot be accessed within the first time period, or the big data cluster cannot be accessed, execute S107.
[0045] In this embodiment, if the big data cluster cannot be accessed within the first time period, or the big data cluster cannot be accessed, for example, when accessing the big data cluster, a direct feedback of an access result indicating that the big data cluster cannot be accessed is given, it indicates that the big data cluster is in a stopped state. The first time period may be set according to the time period for normal access to the big data cluster.
[0046] S104: Process business data of the big data cluster according to the second detection file.
[0047] If the big data cluster can be accessed within the first time period, it can only mean that the big data cluster may be in an operational state. Therefore, in order to further determine whether the big data cluster is in an operational state, in this step, the business data processing detection task is performed on the big data cluster according to the second detection file.
[0048] In this embodiment, the detection task of processing business data on the big data cluster includes multiple test points, and the detection task is to execute the multiple test points in sequence according to a preset order. The second detection file includes execution statements for processing business data based on the big data cluster, specifically: the second detection file includes execution statements corresponding to each test point for testing the big data cluster.
[0049] In this embodiment, the detection task of processing business data of the big data cluster may be to establish a model (i.e., the schame of the big data cluster), establish a data table in the model, write data into the data table, and finally delete the data table. Each test link is regarded as a test point.
[0050] S105: Determine whether the duration of the business data processing is greater than the second duration, or whether the business data processing cannot be performed. If the duration of the business data processing is less than the second duration, execute S106. If the duration of the business data processing is greater than the second duration, or the business data processing cannot be performed, execute S107.
[0051] In this embodiment, the second duration is set according to the duration required for the big data cluster to complete the detection task in an operational state. The big data cluster cannot process business data, specifically, when the big data cluster executes the detection task according to the second detection file, there is a test link that cannot be executed. In this embodiment, it can be known that the big data cluster cannot process business data by feeding back the execution result that cannot be executed.
[0052] S106: Determine that the big data cluster is in operation.
[0053] If the big data set cluster can be accessed within the first time period and the detection task of business data processing is completed within the second time period, it is determined that the big data cluster is in a running state.
[0054] S107. Issue a warning notice.
[0055] In this embodiment, when the big data cluster cannot be accessed, the big data cluster cannot be accessed within a first time period, the duration of business data processing of the big data cluster is greater than the second time period, or the big data cluster is unable to process business data, preset detection result information is generated to characterize that the big data cluster has stopped operating; and the detection result information is output to a detection log file, so that a preset early warning tool reads the detection log file, determines the early warning information corresponding to the detection result information, and sends the early warning information to a preset user communication signal to prompt the big data cluster to stop operating.
[0056] The method provided in this example obtains a detection file of a big data cluster, and the detection file includes at least a first detection file and a second detection file; the first detection file includes access information for accessing the big data cluster; the second detection file includes execution statements for performing business data processing based on the big data cluster; the big data cluster is accessed according to the first detection file, and if the big data cluster cannot be accessed or cannot be accessed within a first time period, it means that the big data cluster has stopped running, and an early warning notification is issued, so that the problem of the big data cluster stopping running can be discovered in time; if the big data cluster is accessed within the first time period, but business data is processed for the big data cluster according to the second detection file; if the time period for business data processing is greater than the second time period, or business data processing cannot be performed, it means that the big data cluster has stopped running, and an early warning notification is issued, so that the problem of the big data cluster stopping running can be discovered in time. In summary, the method provided in this application can discover the problem of the big data cluster stopping running in time.
[0057] Considering that when the big data cluster is in operation, it is very likely that only some nodes in the cluster are running, and some nodes are in a downtime state. Therefore, the detection file described in the above embodiment may also include a third detection file, the third detection file includes an execution statement for detecting the operation state of each node in the big data cluster, and the third detection file is used to detect the operation state of all nodes in the big data cluster. After determining that the big data cluster is in an operation state, the third detection file may be used to detect the operation state of each node in the big data cluster to determine whether there is a downtime node.
[0058] Furthermore, it should be noted that in the embodiment of the present application, as the detection requirements increase, detection files corresponding to the detection requirements can be added. The specific implementation method is: in response to the user input of an instruction to add a new detection file, the new detection file and the identifier carried by the new detection file are obtained, and the corresponding big data cluster of the new detection file is determined based on the identifier.
[0059] The identifier carried by the newly added detection file is used to identify the data cluster corresponding to the detection file. The detection file corresponds to the big data cluster, and the detection file is used to detect the detection cluster. The data cluster corresponding to the detection file can be multiple or single. As the functions of the big data cluster continue to increase, the number of tests that need to be performed also increases, so the corresponding increase in detection files has good practicality and expansibility.
[0060] Furthermore, it should be noted that the detection method for big data clusters provided in this application can be applied to different types of big data clusters, for example, GreenPlum big data clusters and Hadoop big data clusters. Figure 1 The detection method of the big data cluster shown can detect the big data cluster. Furthermore, multiple big data clusters can be detected in parallel. As long as the detection time points preset for each big data cluster are the same, the inspection can be performed according to the detection file of the big data cluster. Similarly, the serial polling method can be used to detect the preset big data clusters in turn.
[0061] The solution provided in this application can detect big data clusters based on the detection files of big data clusters, which is independent of the product type of big data clusters. It can detect different types of big data clusters and has strong adaptability and dynamic scalability.
[0062] Corresponding to the above example, Figure 2 The architecture diagram of a big data cluster detection platform provided for this application example is as follows: Figure 2 As shown, it includes: a collection module, a rule configuration module, a scheduling module, a receiving module, a processing module, and an early warning module. Among them, the collection module runs in the master node of each big data cluster. The big data cluster includes a GreenPlum cluster and a Hadoop cluster.
[0063] The rule configuration module is used to obtain the detection file of the big data cluster, store the detection time point configured by the user, and the big data cluster detected at the detection time point.
[0064] The scheduling module is used to schedule the detection files to be sent to the collection module according to the detection time points in the scheduling module and the big data clusters detected at the detection time points, thereby triggering the collection modules pre-installed on the master nodes of each big data cluster to collect the running status information of the big data cluster.
[0065] The acquisition module detects the big data cluster according to the detection file and returns the detection result to the receiving module. The specific detection method of the acquisition module can refer to the detection method of the big data cluster in the above embodiment. The acquisition module can be a data acquisition agent tool that has the function of querying the collection results and automatically feeding back the collected results.
[0066] The receiving module is used to receive the detection results fed back by the acquisition module.
[0067] The processing module is used to process the test results fed back by the acquisition module so that the test results meet the data format requirements of the early warning module, and output the processed test result information to the test log file.
[0068] The early warning module is used to read the detection log file and determine the early warning information corresponding to the detection result information according to the pre-set early warning rules. After determining that the corresponding early warning information meets the pre-set alarm level threshold, the fault information is sent out through the SMS or email alarm interface, so as to promptly remind the operation and maintenance personnel.
[0069] It should be noted that Figure 2 The connection relationship between the various functional modules of the big data cluster detection platform is only an example. Any detection platform that implements the technical effects of the big data cluster detection platform through the functional modules provided by this application falls within the protection scope of this application.
[0070] The big data cluster detection platform provided in this embodiment can promptly discover the problem of a big data cluster stopping operation, and can detect the big data cluster based on the detection file of the big data cluster. It is independent of the product type of the big data cluster and can detect different types of big data clusters. It has strong adaptability and dynamic scalability.
[0071] Furthermore, no human intervention is required during the detection process, which solves the waste of manpower caused by manual detection in the case of a large number of clusters and avoids the problem of inaccurate detection caused by manual detection due to human subjectivity.
[0072] Figure 3 A schematic diagram of a big data cluster detection device 300 provided in an embodiment of the present application includes:
[0073] The acquisition unit 301 is used to acquire the detection file of the big data cluster, and the detection file includes at least a first detection file and a second detection file; the first detection file includes access information for accessing the big data cluster; the second detection file includes execution statements for business data processing based on the big data cluster.
[0074] The access unit 302 is used to access the big data cluster according to the first detection file, and issue a warning notification if the big data cluster cannot be accessed or cannot be accessed within a first period of time.
[0075] The processing unit 303 is used to perform the business data processing on the big data cluster according to the second detection file if the big data cluster is accessed within the first time period; if the time length of the business data processing is greater than the second time length, or the business data processing cannot be performed, issue the warning notification.
[0076] Optionally, the specific implementation of the access unit 302 and the processing unit 303 issuing the warning notification is:
[0077] When the big data cluster cannot be accessed, the big data cluster cannot be accessed within the first duration, the duration of the business data processing of the big data cluster is greater than the second duration, or the big data cluster cannot perform the business data processing, generating preset detection result information for characterizing that the big data cluster has stopped running;
[0078] Outputting the detection result information to a detection log file;
[0079] The preset early warning tool is triggered to read the detection log file, and send the early warning information corresponding to the detection result information to the preset user communication signal to prompt the big data cluster to stop running.
[0080] Optionally, the specific implementation method of the acquisition unit 301 for acquiring the detection file of the big data cluster is: according to the detection time point specified in the preset detection configuration file, acquiring the detection file of the big data cluster corresponding to the detection time point preset, wherein the detection configuration file includes the corresponding relationship between the detection time point and the big data cluster, and the detection time point corresponds to the big data cluster in that the big data cluster is detected at the detection time point.
[0081] Optionally, the access unit 302 and the processing unit 303 are further used to determine that the big data cluster is in a running state if the duration of the business data processing is less than the second duration.
[0082] Optionally, the access information at least includes:
[0083] The IP address of the big data cluster, the identifier of the big data cluster, and the access password of the big data cluster;
[0084] The specific implementation of the access unit 302 accessing the big data cluster is: based on the access information, accessing the host node of the big data cluster.
[0085] Optionally, the detection file also includes a third detection file, the third detection file includes an execution statement for detecting the operating status of each node in the big data cluster, and the third detection file is used to detect the operating status of all nodes in the big data cluster.
[0086] Optionally, the acquisition unit 301 is also used to, in response to an instruction of a user input to add a new detection file, obtain a new detection file and an identifier carried by the new detection file, and determine the big data cluster corresponding to the new detection file based on the identifier. The detection file corresponds to the big data cluster, so as to use the detection file to detect the big data cluster.
[0087] The device provided in the embodiment of the present application obtains the detection file of the big data cluster, and the detection file includes at least the first detection file and the second detection file; the first detection file includes access information for accessing the big data cluster; the second detection file includes execution statements for performing business data processing based on the big data cluster; the big data cluster is accessed according to the first detection file, and if the big data cluster cannot be accessed or cannot be accessed within a first time period, it means that the big data cluster has stopped running, and an early warning notification is issued, so that the problem of the big data cluster stopping running can be discovered in time; if the big data cluster is accessed within the first time period, but business data processing is performed on the big data cluster according to the second detection file; if the time length of business data processing is greater than the second time period, or business data processing cannot be performed, it means that the big data cluster has stopped running, and an early warning notification is issued, so that the problem of the big data cluster stopping running can be discovered in time. In summary, the method provided in the present application can discover the problem of the big data cluster stopping running in time.
[0088] The present application also provides an electronic device 400, whose structural diagram is as follows: Figure 4 As shown, it includes: a processor 401 and a memory 402, the memory 402 is used to store the application program, and the processor 401 is used to execute the application program to implement the method of big data cluster detection of the present application, that is, to perform the following steps:
[0089] Acquire a detection file of a big data cluster, the detection file comprising at least a first detection file and a second detection file; the first detection file comprises access information for accessing the big data cluster; the second detection file comprises an execution statement for performing business data processing based on the big data cluster;
[0090] Accessing the big data cluster according to the first detection file, and issuing a warning notification if the big data cluster cannot be accessed or cannot be accessed within a first period of time;
[0091] If the big data cluster is accessed within the first time period, the business data of the big data cluster is processed according to the second detection file; if the duration of the business data processing is greater than the second time period, or the business data processing cannot be performed, the early warning notification is issued.
[0092] The present application also provides a computer-readable storage medium, in which instructions are stored. When the computer-readable storage medium is run on a computer, the computer executes the method for detecting big data clusters of the present application, that is, executes the following steps:
[0093] Acquire a detection file of a big data cluster, the detection file comprising at least a first detection file and a second detection file; the first detection file comprises access information for accessing the big data cluster; the second detection file comprises an execution statement for performing business data processing based on the big data cluster;
[0094] Accessing the big data cluster according to the first detection file, and issuing a warning notification if the big data cluster cannot be accessed or cannot be accessed within a first period of time;
[0095] If the big data cluster is accessed within the first time period, the business data of the big data cluster is processed according to the second detection file; if the duration of the business data processing is greater than the second time period, or the business data processing cannot be performed, the early warning notification is issued.
[0096] If the functions described in the method of the embodiment of the present application are implemented in the form of software functional units and sold or used as independent products, they can be stored in a storage medium readable by a computing device. Based on this understanding, the part of the embodiment of the present application that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions to enable a computing device (which can be a personal computer, server, mobile computing device or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.
[0097] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.
[0098] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A method for detecting a big data cluster, characterized in that: include: According to the detection time point specified in the preset detection configuration file, obtain the detection file of the big data cluster preset corresponding to the detection time point, wherein the detection configuration file includes the corresponding relationship between the detection time point and the big data cluster, the detection time point corresponds to the big data cluster in that, at the detection time point, the big data cluster is detected, and the detection file includes at least a first detection file and a second detection file; the first detection file includes access information for accessing the big data cluster; the second detection file includes execution statements for performing business data processing based on the big data cluster; wherein the detection task for performing business data processing on the big data cluster includes multiple test points, the detection task is to execute the multiple test points in sequence according to a preset order, and the second detection file includes execution statements corresponding to the various test points for testing the big data cluster; Accessing the big data cluster according to the first detection file, and issuing a warning notification if the big data cluster cannot be accessed or cannot be accessed within a first period of time; If the big data cluster is accessed within the first time period, the business data processing is performed on the big data cluster according to the second detection file; if the time period of the business data processing is greater than the second time period, or the business data processing cannot be performed, the warning notification is issued; the detection task of performing business data processing on the big data cluster is to establish a model of the big data cluster, establish a data table in the model, write data into the data table, and finally delete the data table, wherein each test link is used as a test point; Wherein, when the detection time points preset for multiple big data clusters are the same, the multiple big data clusters are detected in parallel, or the multiple big data clusters are detected in sequence by serial polling; The detection file also includes a third detection file, which includes an execution statement for detecting the operating status of each node in the big data cluster. The third detection file is used to detect the operating status of all nodes in the big data cluster to determine whether there are any downed nodes in the big data cluster.
2. The method according to claim 1, characterized in that Issuing the aforementioned early warning notice includes: In the case where the big data cluster cannot be accessed, the big data cluster cannot be accessed within the first time period, the duration of the business data processing of the big data cluster is greater than the second time period, or the big data cluster cannot perform the business data processing, generating preset detection result information for characterizing that the big data cluster has stopped running; Outputting the detection result information to a detection log file; The preset early warning tool is triggered to read the detection log file, and send the early warning information corresponding to the detection result information to the preset user communication signal to prompt the big data cluster to stop running.
3. The method according to claim 1, characterized in that It also includes determining that the big data cluster is in a running state if the duration of processing the business data is less than the second duration.
4. The method according to claim 1, characterized in that: The access information at least includes: The IP address of the big data cluster, the identifier of the big data cluster, and the access password of the big data cluster; The accessing the big data cluster includes: accessing a host node of the big data cluster based on the access information.
5. The method according to claim 1, characterized in that Also includes: In response to an instruction input by a user to add a new detection file, obtaining the newly added detection file and an identifier carried by the newly added detection file; The big data cluster corresponding to the newly added detection file is determined according to the identifier. The detection file corresponds to the big data cluster, and the big data cluster is detected by using the detection file.
6. A detection device for a big data cluster, characterized in that: include: An acquisition unit is used to acquire, according to a detection time point specified in a preset detection configuration file, a detection file of a big data cluster preset corresponding to the detection time point, wherein the detection configuration file includes a corresponding relationship between the detection time point and the big data cluster, the detection time point corresponds to the big data cluster in that, at the detection time point, the big data cluster is detected, and the detection file includes at least a first detection file and a second detection file; the first detection file includes access information for accessing the big data cluster; the second detection file includes an execution statement for performing business data processing on the big data cluster; wherein the detection task for performing business data processing on the big data cluster includes a plurality of test points, the detection task is to execute the plurality of test points in sequence according to a preset order, and the second detection file includes an execution statement corresponding to each test point for testing the big data cluster; An access unit is used to access the big data cluster according to the first detection file, and issue a warning notification if the big data cluster cannot be accessed or cannot be accessed within a first period of time; the detection task of processing business data on the big data cluster is to establish a model of the big data cluster, establish a data table in the model, write data into the data table, and finally delete the data table, wherein each test link is used as a test point; a processing unit, configured to, if the big data cluster is accessed within the first time period, perform the business data processing on the big data cluster according to the second detection file; and issue the warning notification if the time period for the business data processing is longer than the second time period, or the business data processing cannot be performed; Wherein, when the detection time points preset for multiple big data clusters are the same, the multiple big data clusters are detected in parallel, or the multiple big data clusters are detected in sequence by serial polling; The detection file also includes a third detection file, which includes an execution statement for detecting the operating status of each node in the big data cluster. The third detection file is used to detect the operating status of all nodes in the big data cluster to determine whether there are any downed nodes in the big data cluster.
7. An electronic device, characterized in that: include: A processor and a memory, wherein the memory is used to store a program; The processor is used to run the program to implement the method for detecting a big data cluster as described in any one of claims 1-5.
8. A storage medium, characterized in that: The storage medium stores instructions, which, when executed on a computer, enable the computer to execute the method for detecting a big data cluster as described in any one of claims 1 to 5.
