Big data platform operation and maintenance method and device based on artificial intelligence agent, and medium

Abstract

A big data platform operation and maintenance method and device based on an artificial intelligence agent, and a medium, relate to the fields of artificial intelligence, agents, and the like. The method comprises: determining, by an agent, a target intention of a target question raised by a target object, the target intention comprising at least one of a diagnosis intention for a big data platform, a data query and analysis intention for the big data platform, and a generation intention for an operation and maintenance suggestion for the big data platform, the big data platform being configured to provide resources to execute jobs; determining, by the agent, a workflow corresponding to the target intention in response to the target intention; executing, by the agent, a subtask in the workflow, and outputting a target result obtained by executing the subtask, the target result comprising at least one of a diagnosis result, a data query and analysis result, and the operation and maintenance suggestion, thereby achieving automatic operation and maintenance of the big data platform and improving the operation and maintenance efficiency of the big data platform. In addition, the method supports multiple target intentions, thereby improving the comprehensiveness of the operation and maintenance of the big data platform.

Description

Technical Field

[0001] This disclosure relates to the fields of artificial intelligence, agents, and intelligent agents, and specifically to a method, apparatus, and medium for operating and maintaining a big data platform based on artificial intelligence intelligent agents. Background Technology

[0002] A big data platform is a platform with capabilities for data collection, storage, computing, and analysis. It aims to solve the challenges of processing massive, high-dimensional, and multi-source heterogeneous data, and provide data-driven decision support for enterprises or organizations.

[0003] In related technologies, enterprises or organizations typically assign personnel to operate and maintain big data platforms. However, this approach requires highly skilled personnel and is relatively inefficient. Summary of the Invention

[0004] This summary section is provided to briefly introduce the concepts, which will be described in detail in the detailed description section below. This summary section is not intended to identify key or essential features of the claimed technical solution, nor is it intended to limit the scope of the claimed technical solution.

[0005] Firstly, this disclosure provides a method for operating and maintaining a big data platform based on artificial intelligence agents, including:

[0006] The intelligent agent determines the target intent of the target question raised by the target object, wherein the target intent includes at least one of the following: diagnostic intent for the big data platform, data query and analysis intent for the big data platform, and intention to generate operation and maintenance suggestions for the big data platform, wherein the big data platform is used to provide resources to perform the operation.

[0007] The agent responds to the target intent and determines a workflow corresponding to the target intent, wherein the workflow includes at least one subtask, the at least one subtask being used to acquire first information related to the target intent and generate a target result for the target intent based on the first information;

[0008] The agent executes the at least one sub-task and outputs the target result obtained from executing the at least one sub-task, wherein the target result includes at least one of the diagnostic result, data query and analysis result, and the operation and maintenance suggestion.

[0009] Secondly, this disclosure provides a big data platform operation and maintenance device based on artificial intelligence intelligent agents, including:

[0010] The first determining module is used to determine the target intent of the target question raised by the target object through the intelligent agent. The target intent includes at least one of the following: a diagnostic intent for the big data platform, a data query and analysis intent for the big data platform, and a generation intent for operation and maintenance suggestions for the big data platform. The big data platform is used to provide resources to perform the operation.

[0011] The second determining module is used to determine a workflow corresponding to the target intent by responding to the target intent through the intelligent agent, wherein the workflow includes at least one subtask, the at least one subtask being used to obtain first information related to the target intent and generate a target result for the target intent based on the first information;

[0012] An output module is configured to execute the at least one subtask through the intelligent agent and output the target result obtained from executing the at least one subtask, wherein the target result includes at least one of diagnostic results, data query and analysis results, and operation and maintenance suggestions.

[0013] Thirdly, this disclosure provides a computer-readable medium having a computer program stored thereon, which, when executed by a processing device, implements the steps of the big data platform operation and maintenance method described in the first aspect.

[0014] Fourthly, this disclosure provides an electronic device, comprising:

[0015] A storage device on which computer programs are stored;

[0016] A processing device is used to execute the computer program in the storage device to implement the steps of the big data platform operation and maintenance method described in the first aspect.

[0017] Fifthly, this disclosure provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the big data platform operation and maintenance method described in the first aspect.

[0018] Through the above technical solution, an intelligent agent is used to obtain the target intent of the target object regarding the target problem raised by the big data platform. Based on the target intent, the intelligent agent determines the corresponding workflow and executes at least one task in the workflow to output the target result. That is, the intelligent agent replaces human intervention to realize the automatic operation and maintenance of the big data platform, thereby improving the efficiency of the big data platform's operation and maintenance. In addition, the target intent includes at least one of the following: diagnostic intent for the big data platform, data query and analysis intent for the big data platform, and intent to generate operation and maintenance suggestions for the big data platform. That is, the intelligent agent can support the processing of operation and maintenance intents of the target object from different dimensions, improving the comprehensiveness of the intelligent agent's operation and maintenance of the big data platform, and ensuring that the big data platform can perform operations stably and reliably.

[0019] Other features and advantages of this disclosure will be described in detail in the following detailed description section. Attached Figure Description

[0020] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale. In the drawings:

[0021] Figure 1 This is a schematic diagram illustrating an application environment for a big data platform operation and maintenance method based on artificial intelligence agents, according to an embodiment of this disclosure.

[0022] Figure 2 This is a flowchart illustrating a big data platform operation and maintenance method based on artificial intelligence agents according to an embodiment of this disclosure;

[0023] Figure 3 This is a timing diagram illustrating a big data platform operation and maintenance method based on artificial intelligence agents according to an embodiment of this disclosure;

[0024] Figure 4 This is a block diagram of a big data platform operation and maintenance device based on an artificial intelligence agent, according to an embodiment of this disclosure;

[0025] Figure 5 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present disclosure. Detailed Implementation

[0026] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0027] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0028] The term "comprising" and its variations as used herein are open-ended inclusions, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below.

[0029] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0030] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0031] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0032] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0033] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the software or hardware, such as the electronic device, application, server, or storage medium performing the operations of this disclosed technical solution, based on the prompt message.

[0034] As an optional but non-limiting implementation, in response to a user's active request, sending a prompt message to the user can be done via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0035] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0036] Meanwhile, it is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and related provisions.

[0037] An AI Agent (Artificial Intelligence Agent) is an entity capable of perceiving its environment, making decisions, and acting autonomously to achieve specific goals. With the continuous development of technology, AI agents are being widely applied.

[0038] Operations and maintenance (O&M) typically refers to the daily management, monitoring, troubleshooting, optimization, and ensuring the stable operation of a big data platform. In related technologies, enterprises or organizations typically assign personnel to manage and maintain the big data platform. However, this approach requires specialized skills and is relatively inefficient.

[0039] In view of the above, this disclosure provides a big data platform operation and maintenance method, apparatus, medium, electronic device and program product based on artificial intelligence intelligent agents. The embodiments of this disclosure will be explained and described below with reference to the accompanying drawings.

[0040] Figure 1 This is a schematic diagram illustrating an application environment for a big data platform operation and maintenance method based on artificial intelligence agents, according to an embodiment of this disclosure. (Refer to...) Figure 1 The application environment includes intelligent agents and big data platforms.

[0041] A big data platform can be structured from top to bottom as follows: cluster management layer, engine layer, storage and scheduling layer, and infrastructure layer. Each layer performs different functions and works together to support the operation of the big data platform. A big data platform can be understood as a data processing platform used for collecting, storing, processing, analyzing, and visualizing massive, diverse, and rapidly generated data.

[0042] The infrastructure layer forms the foundational hardware environment of the big data platform, providing underlying hardware and software support (such as cloud servers and containers) and network support. The storage and scheduling layer is responsible for data storage and resource scheduling. The engine layer can integrate various computing frameworks (i.e., engines) to meet data processing needs in different scenarios, such as offline analysis and real-time computing. The cluster management layer provides cluster creation, resource management, service management, configuration management, node management, and access links, ensuring stable and efficient cluster operation.

[0043] In this embodiment, the big data platform, based on the resources provided by the aforementioned layers, such as clusters, services, and engines, utilizes clusters and services to control the engine and process jobs submitted by the target object. The cluster provides support for the engine and manages and monitors the engine through relevant functions in the cluster management layer, such as engine resource allocation and job scheduling. Jobs, such as data cleaning and data analysis, require execution by specific engines. After receiving a job, the engine requests resources from the cluster according to the job's requirements to execute it. During execution, the job utilizes the computing and storage resources of the big data platform to complete various tasks in parallel or sequentially across multiple nodes. Services provide support for the engine and jobs; for example, storage services provide data storage locations for the engine and jobs, such as storing job input data and output results in corresponding locations; and scheduling services ensure efficient job operation by rationally allocating resources.

[0044] In this disclosure, an intelligent agent refers to an agent that can autonomously perceive and understand the target intent of a target object in response to a target problem posed by a big data platform, autonomously plan a workflow to achieve that target intent, and execute sub-tasks within the workflow to obtain the target result for the target intent, thereby achieving comprehensive automated operation and maintenance of the big data platform.

[0045] Figure 2 This is a flowchart illustrating an operational method for a big data platform based on an artificial intelligence agent, according to an exemplary embodiment of this disclosure. This method can be applied to electronic devices. Furthermore, the method can be executed by an artificial intelligence agent-based big data platform operational device, which can be implemented using software and / or hardware, and the software and / or hardware can be configured within the electronic device. (Refer to...) Figure 2 The operation and maintenance method of the big data platform based on artificial intelligence agents may include steps 210, 220 and 230.

[0046] In step 210, the target intent of the target object's proposed target question is determined by the intelligent agent. The target intent includes at least one of the following: diagnostic intent for the big data platform, data query and analysis intent for the big data platform, and intention to generate operation and maintenance suggestions for the big data platform. The big data platform is used to provide resources to perform the operation.

[0047] Big data platforms here, for example Figure 1 The big data platform shown here includes intelligent agents such as... Figure 1 The intelligent agent shown is capable of interacting with a target object through a dialogue assistant, thereby obtaining information input from the target object, such as a target question posed by a big data platform.

[0048] The resources provided by a big data platform include hardware, software, and network resources, such as clusters, services, engines, queues, and networks, to support job execution.

[0049] In some embodiments, the intelligent agent may include multiple sub-intelligent agents for handling different types of target intentions. Specifically, when faced with a target intention, step 220 is implemented through a sub-intelligent agent corresponding to the intention type. In this embodiment, each sub-intelligent agent corresponds to one type of target intention. This allows the solution to handle different types of target intentions without updating existing sub-intelligent agents. Instead, it can be configured with sub-intelligent agents corresponding to new types of target intentions, exhibiting high flexibility and scalability. The aforementioned diagnostic intention, data query and analysis intention, and generation intention can refer to different types of target intentions.

[0050] In some embodiments, the diagnostic results may include results obtained from diagnosing according to a corresponding dimension, which may be a cluster dimension, service dimension, or job dimension. The dimension can be specified by the target object through its target intent. If the target object does not specify a dimension, it can be determined by the agent based on its historical interactions (i.e., historical dialogues) with the target object. For example, the dimension determined by the agent could be a dimension with high recent user attention, or a dimension with high historical failure frequency in the big data platform, etc. These can be configured so that the agent selects a dimension with high recent user attention or a dimension with high historical failure frequency. The results obtained from diagnosing according to the aforementioned corresponding dimensions may include operational status, configuration status, and root causes of failures, etc.

[0051] Taking the cluster dimension as an example, the running status can include the health status of the cluster, the health status of the nodes in the cluster, the usage of resources (storage resources or computing resources) in the cluster, etc. The root cause of the failure can refer to the root cause related to the cluster dimension. For example, the root cause of failure A is that a node in the cluster has crashed.

[0052] Taking the service dimension as an example, the running status can refer to the health status of the service, the configuration status can refer to the rationality of the service configuration, and the root cause of the failure can refer to the root cause related to the service dimension. For example, the root cause of failure A is the poor performance of service A.

[0053] Taking the job dimension as an example, the running status can refer to the job's execution status, the configuration status can be the rationality of the job's resource configuration, and the root cause of the failure can refer to the root cause related to the job dimension. For example, the root cause of failure A is that the job is blocked. In some embodiments, the diagnostic results may also include potential risks of the big data platform. For example, potential risks may include a node in the cluster possibly going down, the performance of service A possibly degrading, job A possibly experiencing delays, etc.

[0054] In some embodiments, query and analysis results may include query results for indicators and analysis results of the raw data within the data range corresponding to the indicators. The analysis results may represent results in different dimensions, such as trends, outliers, and distribution of the raw data. The dimensions of the analysis results may be specified by the target object through the target intent.

[0055] In some embodiments, operational recommendations can be based on diagnostic results and / or data query and analysis results. These recommendations can improve problems existing in the big data platform, thereby improving the performance of the big data platform in processing tasks. For example, if a node fails, the recommended operation could be to reassign the jobs assigned to that node to other nodes so that the jobs can be executed normally.

[0056] In some embodiments, the target intent can be obtained by: obtaining second information about the target object and the agent; and obtaining the target intent by having the agent perform intent recognition based on the second information.

[0057] Here, the second information is used to characterize the multi-turn dialogue information between the target object and the agent. The multi-turn dialogue information includes information provided by the target object to the agent, including the target object's response to follow-up questions. The follow-up questions are determined by the agent based on the information provided by the target object to the agent, and the follow-up questions are used to determine the target of the diagnostic intent or to determine the data range of the data query and analysis intent.

[0058] The target here can refer to cluster identifiers, service identifiers, and job identifiers. For example, taking the determination of the target at the service dimension as an example, if the target object raises the target question "Analyze the status of the services of the big data platform," the intelligent agent can output a list of services provided by the big data platform. This list includes identifiers of all services provided by the big data platform for the target object to choose from. The target object can select at least one service identifier from the service list, thus determining the target intent as: to analyze the status of the service corresponding to the selected at least one service identifier. By refining the target to cluster identifiers, service identifiers, and job identifiers, the diagnostic granularity of the big data platform can be improved.

[0059] The data scope here can refer to the data mapped by database identifiers and data table identifiers in the big data platform. Since the target object interacts with the intelligent agent using natural language, the information in natural language may be ambiguous or vague. The indicator pointed out by the target object may exist in multiple data tables or databases. Therefore, the intelligent agent can provide all databases or data tables related to the indicator, allowing the target object to make further selections, thereby improving the accuracy of indicator query and analysis.

[0060] It is understandable that the information provided by the target object to the intelligent agent may also include the target question.

[0061] In some embodiments, the agent may provide an intent recognition model that performs semantic understanding of the second information to achieve intent recognition.

[0062] In step 220, the agent responds to the target intent and determines the workflow corresponding to the target intent. The workflow includes at least one subtask, which is used to acquire first information related to the target intent and generate a target result for the target intent based on the first information.

[0063] Here, at least one subtask may include a subtask of acquiring first information and a subtask of analyzing based on the first information to determine the target result. The first information may include information provided by the intelligent agent and information provided by the big data platform.

[0064] Figure 3 This is a timing diagram illustrating a big data platform operation and maintenance method according to an embodiment of this disclosure, with reference to... Figure 3 An intelligent agent may include an intent recognition module, a task planning module, an execution and invocation module, an invocation object, and a memory module.

[0065] The intent recognition module may include the aforementioned intent recognition model. The intent recognition module performs intent analysis on the target question raised by the target object in order to obtain the target intent.

[0066] The intent recognition module transmits the target intent to the task planning module, and can also transmit multi-turn dialogue information at the same time, so that the task planning module can determine the workflow for the target intent and trigger the execution of the workflow.

[0067] Workflow-related information can include the execution order of all subtasks in the workflow and the results obtained from executing each subtask. By storing the workflow-related information in the memory module, the agent can have long short-term memory, accumulate experience, and optimize the agent's future decisions.

[0068] The invoked object can include functions, interfaces, databases, search engines, etc. The invoked object can provide initial information related to the target intent. This initial information can be provided by the intelligent agent itself or by a big data platform. The method by which the intelligent agent obtains initial information from the big data platform can be referred to the following related embodiments, which will not be elaborated upon here.

[0069] For example, taking cluster-level diagnostic intent as an example, the first piece of information can include information from different dimensions within the cluster. This information may include cluster information, cluster resource information, cluster service information, and overall storage and computing information. Cluster information may include basic cluster information (such as cluster identifier), cluster type, and whether the cluster is highly available. High availability refers to whether the cluster can still provide services normally when some nodes fail. Cluster resource information may include node information (such as the number of nodes) and resource usage in the cluster. Resource usage may refer to CPU utilization, memory usage, etc., used to understand the allocation and consumption of cluster resources. Cluster service information may include the service status of services in the cluster, such as normal operation and abnormal operation. Overall storage and computing information involves the overall storage and computing status of the cluster, used to comprehensively control the cluster's operating status at the storage and computing levels. Storage status includes total storage capacity, used capacity, and storage performance (such as read and write speeds). Computing status includes job execution status, computing resource utilization, and job response time.

[0070] For example, taking the diagnostic intent of the service dimension as an example, the first information may include information of different dimensions associated with the service. This information of different dimensions may include cluster information, service basic information, service configuration information, service resource usage and service log information. Cluster information can be found in the explanations and descriptions of the relevant embodiments above; basic service information, such as the service status and the components included in the service, is used to understand whether the service is in a normal operating state and the components included in the service, making it easier to grasp the basic situation of the service as a whole; service configuration information records various configuration parameters that the service depends on, such as the service port configuration, connection pool size configuration, cache-related configuration, business logic-related parameter configuration, etc. These configurations directly affect the functionality and performance of the service, and viewing the configuration information helps to troubleshoot service problems caused by improper configuration; service resource usage refers to the monitoring data of resources used during service operation, such as CPU utilization, memory usage, disk read / write status, network bandwidth usage, etc.; service log information refers to various logs generated during service operation, such as error logs, warning logs, and information logs. The logs record the detailed process of service operation, abnormal situations that occur, etc., and are an important basis for troubleshooting service failures and analyzing service operation behavior. By analyzing the logs, the root cause of service problems can be located.

[0071] For example, taking the diagnostic intent at the job dimension as an example, the first information may include information from different dimensions associated with the job. This information may include, for example, basic job information, basic service information of the job-related services, service configuration information of the job-related services, and resource information of the resources used by the job. Basic job information includes, for example, the structured query statement executed by the job, the job type, and the job execution time. This information allows for a quick understanding of the job's basic characteristics and execution efficiency. Basic service information and service configuration information can be found in the aforementioned related embodiments, and will not be elaborated upon here. Resource information regarding the resources used by the job involves the resource usage during execution, such as CPU resources, memory resources, disk read / write resources, and network resources. This information allows for the assessment of the job's resource consumption level, the identification of resource bottlenecks, and provides a basis for optimizing job resource allocation and improving job execution efficiency.

[0072] For example, taking potential risks as an example, potential risks could include predicting computing resource trends or predicting elasticity patterns. The primary information corresponding to computing resource trends could include basic computing information and computing task information. Elasticity patterns refer to the ability of a big data platform to automatically increase or decrease computing and storage resources based on real-time job demands. The primary information corresponding to the prediction of elasticity patterns could include data related to storage media. Taking a distributed file system as an example, data related to stored files includes the basic information of the distributed file system, the distribution of files by size, and the distribution of hot and cold data. Taking a database as an example, data related to stored files includes the basic information of the database, database information, the distribution of files by size, the distribution of hot and cold data, and the distribution of data table storage formats.

[0073] For example, taking the data query and analysis intent of a big data platform as an example, the first piece of information can be the database and the data table related to the target intent in the big data platform.

[0074] For example, taking the intention of generating operation and maintenance suggestions for a big data platform as an example, the first piece of information could be the root cause of the fault, etc.

[0075] In some embodiments, since the environment in which jobs are performed in the big data platform is dynamic, developers can configure the agent to periodically respond to the target object's target intent according to a preset period, so that the target result can adapt to the dynamic changes in the environment, making it easier for the target object to understand the target result that adapts to the dynamic changes in the environment in a timely manner.

[0076] In step 230, at least one subtask is executed by the intelligent agent, and the target result obtained from executing at least one subtask is output. The target result includes at least one of the following: diagnostic results, data query and analysis results, and operation and maintenance suggestions.

[0077] Continue to refer to Figure 3 The execution and invocation module can return the results of the executed invocation, so that the agent can output the results for the target object to refer to.

[0078] Continue to refer to Figure 3 The intelligent agent can output the workflow and the execution status of each subtask, making it easier for the target object to understand the progress.

[0079] Through the above technical solution, an intelligent agent is used to obtain the target intent of the target object regarding the target problem raised by the big data platform. Based on the target intent, the intelligent agent determines the corresponding workflow and executes at least one task in the workflow to output the target result. That is, the intelligent agent replaces human intervention to realize the automatic operation and maintenance of the big data platform, thereby improving the efficiency of the big data platform's operation and maintenance. In addition, the target intent includes at least one of the following: diagnostic intent for the big data platform, data query and analysis intent for the big data platform, and intent to generate operation and maintenance suggestions for the big data platform. That is, the intelligent agent can support the processing of operation and maintenance intents of the target object from different dimensions, thereby improving the comprehensiveness of the intelligent agent's operation and maintenance of the big data platform.

[0080] In some embodiments, the above-described big data platform operation and maintenance method may further include the following steps: in response to an execution instruction, determining the target operation and maintenance operation corresponding to the operation and maintenance recommendation; and executing the target operation and maintenance operation through an intelligent agent.

[0081] Here, the execution command can be issued by the target object, or it can be actively triggered by the intelligent agent based on the degree of impact of the operation and maintenance on the big data platform.

[0082] When the target object issues the execution instruction, the intelligent agent can provide query information to ask the target object whether to execute the target operation. The target object can respond based on the query information, so that the target object can decide whether to execute the target operation.

[0083] In cases where an intelligent agent proactively triggers execution instructions based on the impact of a target maintenance operation on the big data platform, the agent can determine the degree of impact based on the scope of the target maintenance operation's influence on the big data platform. For target maintenance operations with an impact less than a preset level, the agent can automatically generate execution instructions without manual confirmation to achieve the execution of the target maintenance operation. For target maintenance operations with an impact greater than or equal to the preset level, the agent needs manual confirmation before generating execution instructions.

[0084] In the case where the intelligent agent actively triggers the execution command based on the degree of impact of the target operation on the big data platform, the intelligent agent can learn the user's historical feedback behavior in response to the target operation. That is, the intelligent agent can automatically determine whether to perform the target operation based on the user's historical feedback in response to the target operation and the real environment of the big data platform when the user made the historical feedback. It can be understood that the historical feedback is used to characterize whether the user performs the target operation.

[0085] In some embodiments, the intelligent agent can execute the target operation and maintenance operation in the simulation environment of the big data platform, and determine whether to apply the target operation and maintenance operation to the real environment of the big data platform based on the execution result of the target operation and maintenance operation in the simulation environment.

[0086] The simulation environment here is a simulation of the real environment of the big data platform. The execution results of the target operation in the simulation environment can characterize the performance comparison of the big data platform before and after the execution of the target operation. Therefore, the intelligent agent can determine whether to apply the target operation to the real environment of the big data platform based on the performance changes before and after the execution of the target operation, so as to avoid irreversible impacts on the big data platform caused by directly executing the target operation in the real environment.

[0087] In some embodiments, the target operation and maintenance activity may include at least one of the following:

[0088] Node isolation is used to mark a node when it fails or becomes abnormal, preventing that node from being scheduled to perform jobs and preventing the failure of that node from affecting the cluster it belongs to.

[0089] The fault disk replacement operation is used to replace the storage medium that stores data when it is physically damaged, in order to prevent data loss and service interruption.

[0090] Service configuration tuning operations are used to resolve performance bottlenecks caused by insufficient or unreasonable resource utilization;

[0091] Small file optimization is used to reduce the number of small files, reduce the pressure on metadata management in distributed storage systems, and improve data read and write efficiency.

[0092] Data table bucketing is a process that divides a large data table into multiple sub-tables. This allows the cluster to schedule multiple nodes to perform parallel processing of data based on the data tables.

[0093] Index optimization involves creating indexes on specific columns in a table, enabling the database to quickly locate rows that meet query conditions without scanning the entire table, thereby enhancing database performance.

[0094] In some embodiments, the aforementioned intelligent agent can obtain first information related to the target intent from the big data platform in the following ways: the intelligent agent communicates with the cluster management layer to obtain first access information, which includes the access address of the target engine in the engine layer, the target engine being determined based on the target intent; based on the first access information and index information, second access information is generated, the index information being determined based on the target intent; the intelligent agent sends an acquisition request to the target engine based on the access address in the second access information, the acquisition request being used to acquire the first information related to the target intent corresponding to the index information; and the first information returned by the target engine is received.

[0095] The cluster management layer stores the access addresses of each engine. Agents can obtain these engine access addresses by communicating with the cluster management layer. Both the target engine and index information are determined based on the target intent. The target engine manages primary information related to the target intent, and the index information is used to map this primary information. For example, the index information represents a certain node. Thus, the primary information of the corresponding node can be obtained; this node is understood to be used to support the engine in executing jobs.

[0096] The request carries the complete address of the first information. In some embodiments, the intelligent agent is developed by an external platform equivalent to a big data platform. Therefore, the target engine returns the first information after determining that the target object has passed authentication based on the access key. The access key can be managed by the cluster management layer and can be carried in the first access information so as to be returned to the intelligent agent at the same time as the access address.

[0097] In some embodiments, the above-described big data platform operation and maintenance method may further include the following steps: obtaining the evaluation result of the target result from the memory module in the agent, wherein the evaluation result is used to characterize whether the target result is correct; if the target result is incorrect, taking the target problem and the target result as negative samples; if the target result is correct, taking the target problem and the target result as positive samples; and iterating the agent based on the negative samples and positive samples.

[0098] Continue to refer to Figure 3 The memory module in the intelligent agent stores the feedback of the target object to the target result, which is the evaluation result.

[0099] By automatically constructing negative and positive samples and using them to iterate the agent, the agent can continuously reflect on itself and learn, thereby improving its performance.

[0100] In some embodiments, classification is performed based on negative and positive samples according to intent. The positive and negative samples in the classification results corresponding to the intent are used to iterate the sub-agents in the agent that implement the corresponding intent, thereby accelerating the convergence of the sub-agents.

[0101] In some embodiments, the first information may include third information. The above-mentioned big data platform operation and maintenance method may further include the following steps: creating and executing an offline task for a target intent, the offline task being used to collect third information for the target intent, the third information including the input information and output information of the agent; storing the third information in the memory module of the agent, the first information on which the agent generates the next target result includes the third information, and the target intent corresponding to the next target result is the same as the target intent corresponding to the target question.

[0102] Following the example above, the input information of the agent can be the information provided by the target object to the agent, and the output information of the agent can be the information involved in the process of the agent obtaining the target result by processing the same target intention in the past. For example, it can be the above-mentioned workflow, the results obtained by executing each sub-task, and the feedback from the target object, etc.

[0103] Storing third-party information in the agent's memory module allows the agent to learn from timely interactions with the target object and optimize its future behavior.

[0104] In some embodiments, when the agent determines that the target intent of the next target problem is the same as the historical target intent, it can obtain third information corresponding to the historical target intent from the memory module, so that the agent can combine historical experience to generate a target result for the next target problem.

[0105] Here, the third information included in the first information upon which the agent relies to generate the next target result can be filtered, thereby reducing the amount of data the agent needs to process. For example, in the memory module, when different third information corresponds to the same historical target intent, filtering can be performed in the following way:

[0106] The third piece of information for the candidates is determined, and the historical target intent corresponding to the third piece of information of the candidates is the same as the target intent of the current target problem.

[0107] The target third information in the candidate third information is used as the third information on which the target result of the current target problem depends. The target third information includes the latest third information stored in the memory module from the candidate third information, or the third information corresponding to the target result whose evaluation result is correct from the candidate third information.

[0108] In some embodiments, when the third information of the first target is different from the third information of the second target and the evaluation results corresponding to the third information of the first target and the third information of the second target are respectively, one of the third information of the first target and the third information of the second target can be selected as the target third information according to the priority pre-configured by the target object. The third information of the first target can be the third information that is most recently stored in the memory module among the candidate third information, and the third information of the second target can be the third information corresponding to the target result whose evaluation result is correct among the candidate third information.

[0109] In some embodiments, the agent can also perform knowledge-based question answering for big data platforms. The target object asks a question, such as how the engine is used in the big data platform, and the agent provides the target object with knowledge about engine usage through knowledge base retrieval.

[0110] Figure 4 This is an operation and maintenance device for a big data platform based on an artificial intelligence agent, as shown in the embodiments of this disclosure, with reference to... Figure 4 The big data platform operation and maintenance device 400 based on artificial intelligence agents may include:

[0111] The first determining module 401 is used to determine the target intent of the target question raised by the target object through the intelligent agent, wherein the target intent includes at least one of the following: diagnostic intent for the big data platform, data query and analysis intent for the big data platform, and intention to generate operation and maintenance suggestions for the big data platform, wherein the big data platform is used to provide resources to perform the operation.

[0112] The second determining module 402 is used to determine a workflow corresponding to the target intention through the intelligent agent responding to the target intention, wherein the workflow includes at least one subtask, the at least one subtask being used to obtain first information related to the target intention and generate a target result for the target intention based on the first information;

[0113] The output module 403 is used to execute the at least one sub-task through the intelligent agent and output the target result obtained by executing the at least one sub-task, wherein the target result includes at least one of the diagnostic result, data query and analysis result and the operation and maintenance suggestion.

[0114] Optionally, the target intent can be obtained through the following methods:

[0115] The second information of the target object and the intelligent agent is obtained, wherein the second information is used to characterize the multi-turn dialogue information between the target object and the intelligent agent, the multi-turn dialogue information includes information provided by the target object to the intelligent agent, the information includes the response of the target object to follow-up questions, the follow-up questions are determined by the intelligent agent based on the information provided by the target object to the intelligent agent, and the follow-up questions are used to determine the target of the diagnostic intent or to determine the data range of the data query and analysis intent;

[0116] The agent performs intent recognition based on the second information to obtain the target intent.

[0117] Optionally, the big data platform operation and maintenance device 400 based on artificial intelligence agents further includes:

[0118] The response module is used to respond to the execution command and determine the target maintenance operation corresponding to the maintenance suggestion;

[0119] The execution module is used to execute the target operation and maintenance operation through the intelligent agent.

[0120] Optionally, the big data platform includes a cluster management layer and an engine layer, and the intelligent agent obtains first information related to the target intent from the big data platform in the following ways:

[0121] The agent communicates with the cluster management layer to obtain first access information, wherein the first access information includes the access address of the target engine in the engine layer, and the target engine is determined based on the target intent;

[0122] Based on the first access information and the index information, second access information is generated, wherein the index information is determined based on the target intent;

[0123] The agent sends an acquisition request to the target engine based on the access address in the second access information, wherein the acquisition request is used to acquire first information corresponding to the index information;

[0124] Receive the first information returned by the target engine.

[0125] Optionally, the big data platform operation and maintenance device 400 based on artificial intelligence agents further includes:

[0126] An acquisition module is used to acquire an evaluation result of the target result from a memory module in the agent, wherein the evaluation result is used to characterize whether the target result is correct;

[0127] The third determining module is used to determine the target problem and the target result as negative samples when the target result is incorrect;

[0128] The fourth determining module is used to determine the target problem and the target result as positive samples if the target result is correct.

[0129] An iteration module is used to iterate the agent based on the negative samples and the positive samples.

[0130] Optionally, the big data platform operation and maintenance device 400 based on artificial intelligence agents further includes:

[0131] A creation module is used to create and execute an offline task for the target intent, wherein the offline task is used to collect third information for the target intent, the third information including the input information of the agent and the output information of the agent;

[0132] A storage module is used to store the third information into the memory module of the agent, wherein the first information on which the agent generates the next target result includes the third information, and the target intent corresponding to the next target result is the same as the target intent corresponding to the target question.

[0133] The implementation methods of each module in the above-mentioned big data platform operation and maintenance device 400 based on artificial intelligence agents can refer to the above-mentioned related embodiments, and will not be repeated here.

[0134] This disclosure also provides a computer-readable medium having a computer program stored thereon, which, when executed by a processing device, implements the steps of the above-described big data platform operation and maintenance method based on artificial intelligence agents.

[0135] This disclosure also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the above-described big data platform operation and maintenance method based on artificial intelligence agents.

[0136] This disclosure also provides an electronic device, including:

[0137] A storage device on which computer programs are stored;

[0138] A processing device is used to execute the computer program in the storage device to implement the steps of the above-described big data platform operation and maintenance method based on artificial intelligence agents.

[0139] The following is for reference. Figure 5This diagram illustrates a structural schematic of an electronic device 500 suitable for implementing embodiments of the present disclosure. The terminal devices in these embodiments may include, but are not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 5 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0140] like Figure 5 As shown, electronic device 500 may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 501, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 502 or a program loaded from storage device 508 into random access memory (RAM) 503. RAM 503 also stores various programs and data required for the operation of electronic device 500. Processing unit 501, ROM 502, and RAM 503 are interconnected via bus 504. Input / output (I / O) interface 505 is also connected to bus 504.

[0141] Typically, the following devices can be connected to I / O interface 505: input devices 506 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 507 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 508 including, for example, magnetic tapes, hard disks, etc.; and communication devices 509. Communication device 509 allows electronic device 500 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 5 An electronic device 500 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0142] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 509, or installed from a storage device 508, or installed from a ROM 502. When the computer program is executed by the processing device 501, it performs the functions defined in the methods of embodiments of this disclosure.

[0143] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0144] In some implementations, electronic devices can communicate using any currently known or future-developed network protocol, such as HTTP (Hypertext Transfer Protocol), and can interconnect with digital data communications (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0145] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0146] The aforementioned computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to: determine, through an intelligent agent, the target intent of a target question raised by a target object, wherein the target intent includes at least one of a diagnostic intent for a big data platform, a data query and analysis intent for the big data platform, and a maintenance suggestion generation intent for the big data platform, the big data platform being used to provide resources to perform tasks; determine, through the intelligent agent, a workflow corresponding to the target intent in response to the target intent, wherein the workflow includes at least one subtask, the at least one subtask being used to acquire first information related to the target intent and generate a target result for the target intent based on the first information; execute the at least one subtask through the intelligent agent and output the target result obtained from executing the at least one subtask, wherein the target result includes at least one of a diagnostic result, a data query and analysis result, and the maintenance suggestion.

[0147] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including but not limited to object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0148] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0149] The modules described in the embodiments of this disclosure can be implemented in software or hardware. The names of the modules do not necessarily limit the module itself; for example, the first determining module can also be described as "a module that determines the target intent of a target object's question through an intelligent agent."

[0150] The functions described above in this document can be performed at least in part by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), and so on.

[0151] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0152] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0153] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0154] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative forms of implementing the claims. Regarding the apparatus in the above embodiments, the specific manner in which the various modules perform their operations has been described in detail in the embodiments relating to the method, and will not be elaborated upon here.

Claims

1. A method for operating and maintaining a big data platform based on artificial intelligence agents, characterized in that, include: The intelligent agent determines the target intent of the target question raised by the target object, wherein the target intent includes at least one of the following: diagnostic intent for the big data platform, data query and analysis intent for the big data platform, and intention to generate operation and maintenance suggestions for the big data platform, wherein the big data platform is used to provide resources to perform the operation. The agent responds to the target intent and determines a workflow corresponding to the target intent, wherein the workflow includes at least one subtask, the at least one subtask being used to acquire first information related to the target intent and generate a target result for the target intent based on the first information; The agent executes the at least one sub-task and outputs the target result obtained from executing the at least one sub-task, wherein the target result includes at least one of the diagnostic result, data query and analysis result, and the operation and maintenance suggestion; The target intent is obtained through the following method: acquiring second information between the target object and the intelligent agent, wherein the second information is used to characterize multi-turn dialogue information between the target object and the intelligent agent, the multi-turn dialogue information including information provided by the target object to the intelligent agent, the information including the target object's response to follow-up questions, the follow-up questions being determined by the intelligent agent based on the information provided by the target object to the intelligent agent, and the follow-up questions being used to determine the target of the diagnostic intent or to determine the data range of the data query and analysis intent; and obtaining the target intent through intent recognition by the intelligent agent based on the second information. The big data platform includes a cluster management layer and an engine layer. The intelligent agent obtains first information related to the target intent from the big data platform in the following ways: The intelligent agent communicates with the cluster management layer to obtain first access information, wherein the first access information includes the access address of the target engine in the engine layer, and the target engine is determined based on the target intent; based on the first access information and index information, second access information is generated, wherein the index information is determined based on the target intent; the intelligent agent sends an acquisition request to the target engine based on the access address in the second access information, wherein the acquisition request is used to acquire the first information corresponding to the index information; and the first information returned by the target engine is received.

2. The big data platform operation and maintenance method according to claim 1, characterized in that, Also includes: In response to the execution command, determine the target maintenance operation corresponding to the maintenance recommendation; The target operation and maintenance is performed by the intelligent agent.

3. The big data platform operation and maintenance method according to claim 1, characterized in that, Also includes: The evaluation result of the target result is obtained from the memory module in the agent, wherein the evaluation result is used to characterize whether the target result is correct; If the target result is incorrect, the target problem and the target result will be identified as negative samples. If the target result is correct, the target problem and the target result are identified as positive samples; The agent is iterated based on the negative samples and the positive samples.

4. The big data platform operation and maintenance method according to claim 1, characterized in that, Also includes: Create and execute an offline task for the target intent, wherein the offline task is used to collect third information for the target intent, the third information including the input information of the agent and the output information of the agent; The third information is stored in the memory module of the agent, wherein the first information on which the agent generates the next target result includes the third information, and the target intent corresponding to the next target result is the same as the target intent corresponding to the target question.

5. A big data platform operation and maintenance device based on artificial intelligence intelligent agents, characterized in that, include: The first determining module is used to determine the target intent of the target question raised by the target object through the intelligent agent. The target intent includes at least one of the following: a diagnostic intent for the big data platform, a data query and analysis intent for the big data platform, and a generation intent for operation and maintenance suggestions for the big data platform. The big data platform is used to provide resources to perform the operation. The second determining module is used to determine a workflow corresponding to the target intent by responding to the target intent through the intelligent agent, wherein the workflow includes at least one subtask, the at least one subtask being used to obtain first information related to the target intent and generate a target result for the target intent based on the first information; An output module is configured to execute the at least one sub-task through the intelligent agent and output the target result obtained from executing the at least one sub-task, wherein the target result includes at least one of diagnostic results, data query and analysis results, and operation and maintenance suggestions; The target intent is obtained through the following method: acquiring second information between the target object and the intelligent agent, wherein the second information is used to characterize multi-turn dialogue information between the target object and the intelligent agent, the multi-turn dialogue information including information provided by the target object to the intelligent agent, the information including the target object's response to follow-up questions, the follow-up questions being determined by the intelligent agent based on the information provided by the target object to the intelligent agent, and the follow-up questions being used to determine the target of the diagnostic intent or to determine the data range of the data query and analysis intent; and obtaining the target intent through intent recognition by the intelligent agent based on the second information. The big data platform includes a cluster management layer and an engine layer. The intelligent agent obtains first information related to the target intent from the big data platform in the following ways: The intelligent agent communicates with the cluster management layer to obtain first access information, wherein the first access information includes the access address of the target engine in the engine layer, and the target engine is determined based on the target intent; based on the first access information and index information, second access information is generated, wherein the index information is determined based on the target intent; the intelligent agent sends an acquisition request to the target engine based on the access address in the second access information, wherein the acquisition request is used to acquire the first information corresponding to the index information; and the first information returned by the target engine is received.

6. A computer-readable medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processing device, it implements the steps of the big data platform operation and maintenance method according to any one of claims 1-4.

7. An electronic device, characterized in that, include: A storage device on which computer programs are stored; A processing device is configured to execute the computer program in the storage device to implement the steps of the big data platform operation and maintenance method according to any one of claims 1-4.

8. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the big data platform operation and maintenance method according to any one of claims 1-4.

Images