Data detection method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a data detection method and device, electronic equipment and a storage medium, and the method comprises the steps: storing event data obtained from a container platform to a distributed search cluster, receiving a detection request for the container platform, searching target event data corresponding to the detection request in the distributed search cluster, and sending the target event data to a server; and submitting the target event data, a preset question template and a historical abnormal event list to a pre-trained language model to obtain an anomaly detection result, and performing anomaly processing on the container platform according to the anomaly detection result. The event data generated by the container platform is collected and processed in real time, the natural language understanding ability of the pre-training language model is utilized, the question template and the historical abnormal event list are combined, intelligent abnormal detection is achieved, manual intervention is reduced, the accuracy and efficiency of abnormal detection are improved, abnormal events are processed in time, and the user experience is improved. And the stability and reliability of the container platform are further improved.

Description

Technical Field

[0001] The present application belongs to the field of data processing technology, and specifically relates to a data detection method, a data detection device, an electronic device, and a computer-readable storage medium. Background Art

[0002] Kubernetes (K8S for short) is an open source container platform for automating the deployment, expansion and management of containerized applications. It can manage and orchestrate multiple containers to ensure that they run correctly on the nodes in the cluster. As more and more companies migrate services to Kubernetes, the scale and complexity of container clusters continue to increase. In order to continuously and stably provide container cloud services, anomaly detection of container clusters is crucial.

[0003] At present, the operation and maintenance mode of traditional container clusters usually relies on the experience of operation and maintenance personnel, and uses manual processing to detect and maintain data anomalies. This has high operation and maintenance costs and slow operation and maintenance responses. It is difficult to adapt to the rapidly changing cloud-native environment, affecting the efficiency of container cluster anomaly detection and further affecting the stability of container services. Summary of the invention

[0004] In view of this, the present invention aims to propose a data detection method, device, electronic device and storage medium to solve the problem that the current container cluster uses manual processing to perform data anomaly detection and anomaly maintenance, which affects the efficiency of container cluster anomaly detection and further affects the stability of container services.

[0005] According to a first aspect of the present invention, a data detection method is provided, the method involving a container platform and a distributed search cluster in communication connection, the method comprising:

[0006] storing the event data obtained from the container platform into the distributed search cluster;

[0007] receiving a detection request for the container platform, and searching the distributed search cluster for target event data corresponding to the detection request;

[0008] Submit the target event data, a preset question template, and a list of historical abnormal events to a pre-trained language model to obtain an abnormality detection result;

[0009] The container platform performs exception processing according to the exception detection result.

[0010] Optionally, storing the event data obtained from the container platform to the distributed search cluster includes:

[0011] Monitoring the container platform through a monitoring interface pre-deployed on the container platform;

[0012] Acquire event data generated by the container platform, and convert the current data format of the event data into the data format of a distributed search cluster;

[0013] The format-converted event data is stored in the distributed search cluster.

[0014] Optionally, after storing the format-converted event data in the distributed search cluster, the method further includes:

[0015] Determine event information corresponding to the event data; the event information includes at least one of event type, event occurrence time, and event level;

[0016] classifying the event data in the distributed search cluster according to the event information;

[0017] An index is created for the classified event data according to the event information.

[0018] Optionally, the receiving a detection request for the container platform, and searching the distributed search cluster for target event data corresponding to the detection request, includes:

[0019] Predetermining detection conditions for the container platform; wherein the detection conditions include a detection period and a detection target, the detection target includes event information to be retrieved, and the event information includes at least one of an event type, an event occurrence time, and an event level;

[0020] Receiving a detection request for the distributed search cluster generated according to the detection condition;

[0021] Sending the detection request to the distributed search cluster regularly using the detection period;

[0022] In response to the detection request being sent to the distributed search cluster, using the detection target to perform data search in the distributed search cluster to determine target event data corresponding to the detection target;

[0023] The target event data corresponding to the detection target is arranged using a preset arrangement strategy to obtain the target event data corresponding to the detection request.

[0024] Optionally, submitting the target event data, a preset question template and a historical abnormal event list to a pre-trained language model to obtain an abnormality detection result includes:

[0025] Obtaining a preset question template and a pre-generated historical abnormal event list; the historical abnormal event list includes abnormal event data detected in a historical detection cycle;

[0026] Determine the event information of the target event data, and fill the event information into the question template;

[0027] Packing the target event data, the filled question template and the historical abnormal event list and submitting them to the pre-trained language model;

[0028] The language model is used to perform anomaly detection on the target event data according to the question template and the historical abnormal event list to obtain an anomaly detection result.

[0029] Optionally, performing exception processing on the container platform according to the exception detection result includes:

[0030] Receiving anomaly detection results of the language model on the target event data; wherein the anomaly detection results include common anomaly data or unknown anomaly data;

[0031] The exception detection result is processed by using a processing strategy corresponding to the exception detection result to perform exception processing on the container platform.

[0032] Optionally, the adopting a processing strategy corresponding to the abnormal detection result to perform abnormal processing on the container platform includes:

[0033] If the abnormality detection result is determined to be common abnormal data, the container platform is cleared of abnormalities according to the common abnormal data and the historical abnormal event list;

[0034] If the abnormality detection result is determined to be unknown abnormal data, an abnormality warning prompt of the container platform is generated, and the abnormality warning prompt is sent to the operation and maintenance personnel of the container platform.

[0035] According to a second aspect of the present invention, a data detection device is provided, the device involving a container platform and a distributed search cluster in communication connection, the device comprising:

[0036] A data storage module, used for storing event data obtained from the container platform to the distributed search cluster;

[0037] A data search module, configured to receive a detection request for the container platform, and search for target event data corresponding to the detection request in the distributed search cluster;

[0038] An anomaly detection module, used to submit the target event data, a preset question template and a historical abnormal event list to a pre-trained language model to obtain an anomaly detection result;

[0039] The exception handling module is used to perform exception handling on the container platform according to the exception detection result.

[0040] According to another aspect of the present invention, there is also provided an electronic device, comprising:

[0041] processor;

[0042] a memory for storing instructions executable by the processor;

[0043] The processor is configured to execute the instructions to implement the data detection method as described above.

[0044] According to another aspect of the present invention, a readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the data detection method described above are implemented.

[0045] The data detection method provided by the embodiment of the present invention stores the event data obtained from the container platform in a distributed search cluster, receives a detection request for the container platform, searches for the target event data corresponding to the detection request in the distributed search cluster, submits the target event data and a preset question template and a list of historical abnormal events to a pre-trained language model, obtains anomaly detection results, and performs abnormal processing on the container platform according to the anomaly detection results. The embodiment of the present invention collects and processes the event data generated by the container platform in real time, utilizes the natural language understanding ability of the pre-trained language model, combines the question template and the list of historical abnormal events, realizes intelligent anomaly detection, reduces manual intervention, reduces the time cost of manual troubleshooting, reduces fault handling delays or detection errors caused by manual factors, improves the accuracy and efficiency of anomaly detection, handles abnormal events in a timely manner, and further improves the stability and reliability of the container platform.

[0046] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. Moreover, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0048] Figure 1 is a flow chart of steps of a data detection method provided by an embodiment of the present invention;

[0049] Figure 2 yes Figure 1A flowchart of step 101 in a data detection method provided by an embodiment of the present invention;

[0050] Figure 3 yes Figure 1 A flowchart of step 102 in a data detection method provided by an embodiment of the present invention;

[0051] Figure 4 yes Figure 1 A flowchart of step 103 in a data detection method provided by an embodiment of the present invention;

[0052] Figure 5 yes Figure 1 A flowchart of step 104 in a data detection method provided by an embodiment of the present invention;

[0053] Figure 6 is a scenario schematic diagram of a data detection method provided by an embodiment of the present invention;

[0054] Figure 7 is a structural schematic diagram of a data detection device provided by an embodiment of the present invention;

[0055] Figure 8 It is a structural schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0056] To make the purpose, technical scheme and advantages of the embodiments of the present invention clearer, the embodiments of the present invention will be described in detail below in conjunction with the accompanying drawings. However, it will be appreciated by those skilled in the art that in the embodiments of the present invention, many technical details are proposed in order to enable the reader to better understand the present application. However, even without these technical details and various changes and modifications based on the following embodiments, the technical scheme claimed in the present application can be implemented. The division of the following embodiments is for the convenience of description, and should not constitute any limitation on the specific implementation of the present invention. The various embodiments can be combined and referenced with each other without contradiction.

[0057] Reference Figure 1 , shows a flow chart of the steps of a data detection method provided by an embodiment of the present invention, the method involves a container platform and a distributed search cluster connected in communication, and the method may include:

[0058] Step 101 : store the event data obtained from the container platform into the distributed search cluster.

[0059] In an embodiment of the present invention, in order to solve the problem that the current container cluster uses manual processing to perform data anomaly detection and anomaly maintenance, which affects the efficiency of container cluster anomaly detection and further affects the stability of container services, this embodiment collects and processes event data generated by the container platform in real time, utilizes the natural language understanding ability of the pre-trained language model, combines question templates and historical abnormal event lists, and implements intelligent anomaly detection, reduces manual intervention, reduces the time cost of manual troubleshooting, reduces fault handling delays or detection errors caused by human factors, improves the accuracy and efficiency of anomaly detection, handles abnormal events in a timely manner, and further improves the stability and reliability of the container platform.

[0060] It should be noted that, refer to Figure 6 , showing a scenario diagram of a data detection method provided by an embodiment of the present invention, the container platform is communicatively connected with the distributed search cluster, the container platform can be a container cluster composed of multiple container platforms, this embodiment takes Kubernetes as an example for explanation of the container platform, the execution subject of the embodiment of the present invention can be a server for deploying monitored objects and detection services, wherein the deployed container platform is the monitored object, the distributed search cluster is deployed to provide event data storage and retrieval services, and the open source large language model is deployed to provide reasoning services. The open source large language model is composed of GPU hardware, an open source general large model, and an open source reasoning framework service, which are not described one by one in this embodiment.

[0061] Specifically, the server is deployed with a listening interface on the container platform, which listens to the container platform through the listening interface. When the event data generated by the container platform is listened to, the event data is stored in the distributed search cluster, where the event data is the real-time status changes and operation information of various components and resources in the container cluster. In the Kubernetes container platform, event data is used to record events that occur in the container cluster. Event data can be a record of any action, such as Pod startup, node status change, or deployment update. Each event data contains detailed information about the event that occurred. The event information includes event type, event cause, event level, involved objects, message content, and timestamp of the event generation time.

[0062] It should be noted that the distributed search cluster is used to persist event data. In this embodiment, the distributed search cluster can be an Elasticsearch cluster, where the Elasticsearch cluster is a distributed search and analysis engine composed of multiple nodes that work together to store, retrieve and analyze large amounts of data. Since the event data is stored for a short time within Kubernetes, the ElasticSearch cluster data can save data for a longer period of time. As a centralized data storage, the ElasticSearch cluster supports event data storage for multiple Kubernetes container clusters and can also perform cross-cluster data analysis.

[0063] Step 102: Receive a detection request for the container platform, and search for target event data corresponding to the detection request in the distributed search cluster.

[0064] In an embodiment of the present invention, the server receives a detection request initiated by an operation and maintenance personnel of a container platform or a detection request generated according to a detection condition, and searches a distributed search cluster for target event data corresponding to the detection request based on the detection request for the container platform, where the target event data is event data that satisfies the detection condition corresponding to the detection request.

[0065] Specifically, the server stores the event data in a distributed search cluster. The distributed search cluster creates an index with the event data. Based on the detection request for the container platform, the server searches the distributed search cluster. The server uses the search function of the distributed search cluster to retrieve event data that matches the detection request and determines the target event data corresponding to the detection request. The detection request includes a detection cycle and a detection target. The detection cycle is used to periodically search for event data in the distributed search cluster. The detection target contains event information of the event data that needs to be detected, which will not be described in detail here.

[0066] Step 103 , submit the target event data, the preset question template and the historical abnormal event list to the pre-trained language model to obtain the abnormality detection result.

[0067] In an embodiment of the present invention, the server submits the target event data, a preset question template and a historical abnormal event list to a pre-trained language model to obtain an abnormality detection result, wherein the preset question template is a question template generated for the detection requirements of the event data, and is used to help the language model understand the detection task. The preset question template can be obtained from a pre-made prompt vocabulary. The historical abnormal event list includes abnormal event data detected in the historical detection cycle, including abnormal event data clearly detected in the historical detection cycle and event information and abnormal solution measures corresponding to the abnormal event data. The pre-trained language model in this embodiment can be any large language model, and this embodiment does not specifically limit the training process of the language model.

[0068] Step 104: Perform exception processing on the container platform according to the exception detection result.

[0069] In this embodiment, the anomaly detection result includes whether the event data is abnormal, the anomaly type and the cause analysis. The anomaly type includes common anomaly data and unknown anomaly data. Common anomaly data is a known anomaly event that matches the historical anomaly event list and has a clear processing strategy. Unknown anomaly data is an anomaly event that has not appeared in the historical anomaly event list. If it is determined to be common anomaly data, the container platform is cleared of anomalies based on the common anomaly data and the historical anomaly event list. If it is determined to be unknown anomaly data, an abnormal warning prompt for the container platform is generated and sent to the operation and maintenance personnel of the container platform.

[0070] The data detection method provided by the embodiment of the present invention stores the event data obtained from the container platform in a distributed search cluster, receives a detection request for the container platform, searches for the target event data corresponding to the detection request in the distributed search cluster, submits the target event data and a preset question template and a list of historical abnormal events to a pre-trained language model, obtains anomaly detection results, and performs abnormal processing on the container platform according to the anomaly detection results. The embodiment of the present invention collects and processes the event data generated by the container platform in real time, utilizes the natural language understanding ability of the pre-trained language model, combines the question template and the list of historical abnormal events, realizes intelligent anomaly detection, reduces manual intervention, reduces the time cost of manual troubleshooting, reduces fault handling delays or detection errors caused by manual factors, improves the accuracy and efficiency of anomaly detection, handles abnormal events in a timely manner, and further improves the stability and reliability of the container platform.

[0071] Further, see Figure 2 , showing Figure 1 A flowchart of step 101 in a data detection method is provided. The method is substantially the same as the data detection method provided in the first embodiment of the present invention. Step 101 may include:

[0072] Step 1011: monitor the container platform through a monitoring interface pre-deployed on the container platform.

[0073] Step 1012, obtain the event data generated by the container platform, and convert the current data format of the event data into the data format of the distributed search cluster.

[0074] Step 1013: store the format-converted event data in the distributed search cluster.

[0075] It should be noted that, in an embodiment of the present invention, a listening component is deployed in the container platform as a listening interface. The listening interface is used to capture event data generated by the container platform in real time. The server can listen to events in the container platform in real time through the listening interface, and send the event data to the message queue or directly process it, read the event data from the listening interface or the message queue, and uniformly convert the current data format of the event data into a format supported by the distributed search cluster, and write the converted event data into the distributed search cluster storage.

[0076] The embodiments of the present invention capture events of the container platform in real time through a monitoring interface, convert the event data into a format supported by a distributed search cluster, facilitate subsequent query and analysis, improve the readability and availability of the data, and utilize the high-performance storage and retrieval capabilities of the distributed search cluster to support the processing of large-scale event data.

[0077] Specifically, after step 1013 stores the format-converted event data in the distributed search cluster, the following may also be included:

[0078] First, determine the event information corresponding to the event data; the event information includes at least one of the event type, event occurrence time, and event level;

[0079] Secondly, the event data is classified according to the event information in the distributed search cluster;

[0080] Secondly, create an index for the classified event data according to the event information.

[0081] It should be noted that in the above steps, after the format-converted event data is stored in the distributed search cluster, the event information corresponding to the event data is determined in the distributed search cluster, wherein the event information includes at least one of the event type, event occurrence time, and event level. Specifically, the event type is such as Pod creation, Pod deletion, node failure, etc. The event occurrence time is determined by the timestamp of the event, and the event level is such as normal, warning, error, etc. The event data is classified according to the fields such as event type, event occurrence time, and event level. Specifically, the aggregate query function in Elasticsearch can be used to classify the event data. After classification, different indexes are created according to the event type or event level. The created index is used to improve the efficiency of data retrieval and search in the cluster.

[0082] The embodiment of the present invention classifies and indexes event data through event information, significantly improves query efficiency, supports multi-dimensional analysis of event data by event type, event level, etc., reduces data troubleshooting time through classification and indexing, and further improves operation and maintenance efficiency.

[0083] Further, see Figure 3 , showing Figure 1 A flowchart of step 102 in a data detection method is provided. The method is substantially the same as the data detection method provided in the first embodiment of the present invention. Step 102 may include:

[0084] Step 1021, predetermine the detection conditions for the container platform; wherein the detection conditions include a detection period and a detection target, the detection target includes event information to be retrieved, and the event information includes at least one of an event type, an event occurrence time, and an event level.

[0085] Step 1022, receiving a detection request for a distributed search cluster generated according to the detection condition.

[0086] Step 1023: Send a detection request to the distributed search cluster regularly using a detection period.

[0087] Step 1024, in response to the detection request being sent to the distributed search cluster, the detection target is used to perform data search in the distributed search cluster to determine the target event data corresponding to the detection target.

[0088] Step 1025: Arrange the target event data corresponding to the detection target using a preset arrangement strategy to obtain the target event data corresponding to the detection request.

[0089] It should be noted that, in an embodiment of the present invention, the server predetermines the detection conditions for the container platform, wherein the detection conditions include a detection cycle and a detection target. The detection cycle is a cycle for the server to perform data detection on the container platform. The detection target is used to screen out target detection events to be detected. The detection cycle and the detection target are set according to actual detection requirements and will not be described in detail here.

[0090] Specifically, the detection cycle is such as performing a detection once every minute, and the detection target includes event information to be retrieved, specifically including at least one of the event type, event occurrence time, and event level. For example, it can be to search for event data generated by the container platform in the last 5 minutes, or to search for event data whose event type is Pod node failure in the container platform, etc. The server receives a detection request containing a detection target and a detection cycle generated according to the detection conditions, and sends a detection request to the distributed search cluster regularly using the detection cycle. When it is detected that the detection request is sent to the distributed search cluster, the detection target is used to search for data in the distributed search cluster to determine the target event data corresponding to the detection target. Specifically, the detection request is executed in the distributed search cluster to search for matching target event data, and the target event data corresponding to the detection target is arranged using a preset arrangement strategy to obtain the target event data corresponding to the detection request, wherein the arrangement strategy can be arranged in ascending or descending order by timestamp or by event level priority, which is not specifically limited here.

[0091] For example, the server generates a detection request based on the information contained in the event data, such as the event level and the time when the event occurred, and requests the distributed search cluster to retrieve the event data. The distributed search cluster periodically performs the retrieval task with an execution period of 1 minute, and pulls the event data of the last 5 minutes each time, and assembles the target event data according to the arrangement strategy. It should be noted that this embodiment does not specifically limit the arrangement strategy, which can be any arrangement order. In this embodiment, the target event data is assembled, and the following two arrangement strategies are used as examples for explanation:

[0092] a) Arrange in reverse chronological order. The target event data are arranged as follows:

[0093] The namespace to which the event data belongs is default, the instance name is loading-pod, the event type is Pod, the event cause is FailedScheduling, the event content is XXXX, and the event generation time is 2024-10-12 14:46:48;

[0094] The namespace to which the event data belongs is default, the instance name is calcing-pod, the event type is Pod, the event reason is FailedScheduling, the event content is XXXX, and the time is 2024-10-12 14:46:38.

[0095] b) Group by event type and event cause, find the latest 3 non-repetitive event contents, and arrange the target event data as follows:

[0096] The event type of the event data is Endpoints. The event cause is FailedToUpdateEndpoint. It was generated once in the last 5 minutes. The contents of the latest 3 events are as follows: (1) xxx, (2) xxx, (3) xxx

[0097] The event type of the event data is CronJob, the event cause is TooManyMissedTimes, and it has occurred twice in the last 5 minutes. The contents of the latest 3 events are as follows: (1) xxx, (2) xxx, and (3) xxx.

[0098] The embodiments of the present invention reduce manual intervention and ensure timely discovery of problems by periodically sending detection requests. The high-performance storage and retrieval capabilities of the distributed search cluster are used to support the processing of large-scale event data. Event data is accurately retrieved according to detection targets to improve query performance. The detection conditions and arrangement strategies can be flexibly configured to meet the needs of different scenarios. Through automated detection and precise retrieval, there is no need for manual investigation, thereby improving detection efficiency.

[0099] Further, see Figure 4 , showing Figure 1 A flowchart of step 103 in a data detection method is provided. The method is substantially the same as the data detection method provided in the first embodiment of the present invention. Step 103 may include:

[0100] Step 1031, obtaining a preset question template and a pre-generated historical abnormal event list; the historical abnormal event list includes abnormal event data detected in a historical detection cycle.

[0101] It should be noted that in the embodiment of the present invention, after determining the target event data to be detected, the server obtains a preset question template and a pre-generated historical abnormal event list, wherein the question template is a question content template generated for the event data generated by the container platform, which is used to help the language model understand the detection task, and can be pre-formulated and modified according to the language model and detection requirements. The historical abnormal event list includes abnormal event data detected in the historical detection cycle, including abnormal event data clearly detected in the historical detection cycle, and event information and abnormal resolution measures corresponding to the abnormal event data.

[0102] Specifically, the preset question template includes prompt words, and the question template is obtained from a pre-made prompt word library. For example, the question template can be: "You are a container expert and intelligent operation and maintenance expert, and you are very familiar with the event data in the container platform and the causes of their occurrence. The input information includes two parts: (1) event data generated by the container platform, including event type, cause, number of occurrences within 5 minutes, and detailed error information of the latest 3 events; (2) a list of historical abnormal events, including common container platform problems and information that needs to be clarified. Please detect the abnormal event data and the abnormal cause based on the above information. The event data is as follows: XX;"

[0103] In this embodiment, the historical abnormal event list includes abnormal event data detected in the historical detection cycle, including abnormal event data clearly detected in the historical detection cycle and event information and abnormal solution measures corresponding to the abnormal event data. The historical abnormal event list is generated from abnormal event data and abnormal solution measures extracted from the expert database of common abnormal problems. An example is as follows:

[0104] 1) Exception type: Etcd service exception; Required information: None; Solution: Restart the Etcd service;

[0105] 2) Exception type: APIServer service exception; Required information: None; Solution: Restart its Etcd service.

[0106] 3) Exception type: Host exception; Required type: Abnormal host name and IP address; Solution: Restart the host.

[0107] Step 1032, determine the event information of the target event data, and fill the event information into the question template.

[0108] Step 1033, packaging the target event data, the filled question template, and the historical abnormal event list and submitting them to the pre-trained language model;

[0109] Step 1034, using a language model to perform anomaly detection on the target event data according to the question template and the historical abnormal event list to obtain anomaly detection results.

[0110] Specifically, the event information of the target event data retrieved from the distributed search cluster is filled into the template, and the target event data, the filled question template and the historical abnormal event list are packaged and submitted to the pre-trained language model, that is, the filled question template, the target event data and the historical abnormal event list are packaged and submitted to the language model, and the language model is used to perform anomaly detection on the target event data according to the question template and the historical abnormal event list. The language model detects whether the target event data is abnormal based on the question template and the historical abnormal event list, and outputs the anomaly detection result to the server.

[0111] It should be noted that, in this embodiment, the language model parses the question template and the target event data, understands the requirements of the detection task, and analyzes whether the target event data matches the known abnormal events in combination with the historical abnormal event list. If the match is successful, the abnormal detection result is output. If the match is not successful, the pre-trained knowledge and context information are used to identify the abnormal features in the target event data. Specifically, the language model extracts key features from the target event data, such as event type, event level, etc., and compares the target event data with the event data in the historical abnormal event list. If the target event data is highly similar to the event data in the historical abnormal event list, it is determined to be abnormal event data; if the target event data does not match the event data in the historical abnormal event list, it is further analyzed whether it has abnormal features and generates an abnormal detection result. The abnormal detection result includes whether it is abnormal, the abnormal type (common or unknown) and the cause analysis.

[0112] The embodiment of the present invention utilizes a pre-trained language model to implement intelligent anomaly detection, provides more accurate detection results through question templates and historical abnormal event lists, quickly processes and analyzes event data, provides real-time detection results, and reduces the time cost of manual troubleshooting through automated anomaly detection.

[0113] Further, see Figure 5 , showing Figure 1 A flowchart of step 104 in a data detection method is provided. The method is substantially the same as the data detection method provided in the first embodiment of the present invention. Step 104 may include:

[0114] Step 1041, determining anomaly detection results of the language model on the target event data; wherein the anomaly detection results include common anomaly data or unknown anomaly data.

[0115] Step 1042: Use a processing strategy corresponding to the abnormal detection result to perform abnormal processing on the container platform.

[0116] It should be noted that in the embodiment of the present invention, the server receives the anomaly detection result of the language model on the target event data, wherein the anomaly detection result includes common anomaly data or unknown anomaly data, common anomaly data is a known anomaly event that matches the historical anomaly event list and has a clear processing strategy, and unknown anomaly data is an anomaly event that has not appeared in the historical anomaly event list. Different processing logics are executed according to the anomaly type (common or unknown) to perform exception processing on the container platform.

[0117] Specifically, step 1042 uses a processing strategy corresponding to the abnormal detection result to perform abnormal processing on the container platform, which may specifically include the following steps:

[0118] If the anomaly detection result is determined to be common abnormal data, the container platform will be cleared of abnormalities based on the common abnormal data and the list of historical abnormal events;

[0119] If the anomaly detection result is determined to be unknown abnormal data, an abnormal warning prompt for the container platform is generated and sent to the operation and maintenance personnel of the container platform.

[0120] It should be noted that in the above steps, event data is extracted from the anomaly detection results, its type is confirmed to be a common anomaly, and a processing strategy matching the common anomaly data is searched from the historical anomaly event list; event data is extracted from the anomaly detection results, its type is confirmed to be an unknown anomaly, and an anomaly warning prompt is generated based on the unknown anomaly data. The anomaly warning prompt includes information such as event type, event level, and event generation time. The anomaly warning prompt can be sent to the operation and maintenance personnel of the container platform via email, SMS, instant messaging tools or alarm system.

[0121] The embodiments of the present invention receive anomaly detection results in real time and execute processing strategies, shorten fault recovery time, automatically execute predefined processing operations for common abnormal events, reduce manual intervention, and reduce the workload of operation and maintenance personnel by automatically processing common abnormalities. Alarms are triggered and recorded for unknown abnormal events to remind operation and maintenance personnel to handle them in a timely manner, thereby improving the stability of the container platform.

[0122] Reference Figure 7 , shows a schematic diagram of the structure of a data detection device provided by an embodiment of the present invention, the device involves a container platform and a distributed search cluster connected in communication, and the device includes:

[0123] A data storage module 201, used to store event data obtained from the container platform to the distributed search cluster;

[0124] A data search module 202 is used to receive a detection request for the container platform and search for target event data corresponding to the detection request in the distributed search cluster;

[0125] Anomaly detection module 203, used to submit the target event data, a preset question template and a historical abnormal event list to a pre-trained language model to obtain anomaly detection results;

[0126] The exception handling module 204 is used to perform exception handling on the container platform according to the exception detection result.

[0127] Furthermore, the data storage module 201 includes:

[0128] A monitoring submodule, used to monitor the container platform through a monitoring interface pre-deployed on the container platform;

[0129] A format processing submodule, used to obtain the event data generated by the container platform, and convert the current data format of the event data into the data format of the distributed search cluster;

[0130] The storage submodule is used to store the event data after format conversion in the distributed search cluster.

[0131] Furthermore, the data storage module 201 further includes:

[0132] An information determination submodule, used to determine event information corresponding to the event data; the event information includes at least one of an event type, an event occurrence time, and an event level;

[0133] A classification submodule, used for classifying the event data according to the event information in the distributed search cluster;

[0134] The index submodule is used to create an index for the classified event data according to the event information.

[0135] Furthermore, the data search module 202 includes:

[0136] A determination condition submodule, used to predetermine a detection condition for the container platform; wherein the detection condition includes a detection period and a detection target, the detection target includes event information to be retrieved, and the event information includes at least one of an event type, an event occurrence time, and an event level;

[0137] A receiving submodule, configured to receive a detection request for the distributed search cluster generated according to the detection condition;

[0138] The sending submodule is used to send a detection request to the distributed search cluster regularly using a detection cycle;

[0139] A search submodule is used to respond to a detection request sent to a distributed search cluster, use the detection target to perform data search in the distributed search cluster, and determine the target event data corresponding to the detection target;

[0140] The arrangement submodule is used to arrange the target event data corresponding to the detection target using a preset arrangement strategy to obtain the target event data corresponding to the detection request.

[0141] Furthermore, the anomaly detection module 203 includes:

[0142] An acquisition submodule is used to acquire a preset question template and a pre-generated historical abnormal event list; the historical abnormal event list includes abnormal event data detected in a historical detection cycle;

[0143] A filling submodule, used to determine the event information of the target event data, and fill the event information into the question template;

[0144] A submission submodule, used to package the target event data, the filled question template and the historical abnormal event list and submit them to the pre-trained language model;

[0145] The detection submodule is used to use the language model to perform anomaly detection on the target event data according to the question template and the historical abnormal event list to obtain anomaly detection results.

[0146] Furthermore, the exception handling module 204 includes:

[0147] A determination result submodule is used to determine anomaly detection results of the language model on the target event data; wherein the anomaly detection results include common anomaly data or unknown anomaly data;

[0148] The exception handling submodule is used to adopt the processing strategy corresponding to the exception detection result to perform exception handling on the container platform.

[0149] Furthermore, the exception handling submodule includes:

[0150] A first processing unit, configured to clear the container platform of abnormalities according to the common abnormal data and a list of historical abnormal events if the abnormality detection result is determined to be common abnormal data;

[0151] The second processing unit is used to generate an abnormal warning prompt for the container platform if the abnormal detection result is determined to be unknown abnormal data, and send the abnormal warning prompt to the operation and maintenance personnel of the container platform.

[0152] The data detection device provided by the embodiment of the present invention stores the event data obtained from the container platform in a distributed search cluster, receives a detection request for the container platform, searches for the target event data corresponding to the detection request in the distributed search cluster, submits the target event data and a preset question template and a list of historical abnormal events to a pre-trained language model, obtains anomaly detection results, and performs abnormal processing on the container platform according to the anomaly detection results. The embodiment of the present invention collects and processes the event data generated by the container platform in real time, utilizes the natural language understanding ability of the pre-trained language model, combines the question template and the list of historical abnormal events, realizes intelligent anomaly detection, reduces manual intervention, reduces the time cost of manual troubleshooting, reduces fault handling delays or detection errors caused by manual factors, improves the accuracy and efficiency of anomaly detection, handles abnormal events in a timely manner, and further improves the stability and reliability of the container platform.

[0153] Reference Figure 8 , an embodiment of the present invention further provides an electronic device, such as Figure 8 As shown, it includes a processor 301, a communication interface 302, a memory 303 and a communication bus 304, wherein the processor 301, the communication interface 302, and the memory 303 communicate with each other through the communication bus 304.

[0154] A processor 301, a memory 303 for storing processor executable instructions;

[0155] The processor 301 is configured to execute the instructions to implement the data detection method as described above.

[0156] The communication bus mentioned in the above terminal can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0157] The communication interface is used for communication between the above terminal and other devices.

[0158] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0159] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0160] In another embodiment of the present invention, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the data detection method described in any of the above embodiments is implemented.

[0161] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk Solid State Disk (SSD)), etc.

[0162] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0163] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0164] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A data detection method, characterized in that: The method involves a container platform and a distributed search cluster that are in communication connection, and the method includes: storing the event data obtained from the container platform into the distributed search cluster; receiving a detection request for the container platform, and searching the distributed search cluster for target event data corresponding to the detection request; Submit the target event data, a preset question template, and a list of historical abnormal events to a pre-trained language model to obtain an abnormality detection result; The container platform performs exception processing according to the exception detection result.

2. The method according to claim 1, characterized in that: The storing of the event data obtained from the container platform to the distributed search cluster includes: Monitoring the container platform through a monitoring interface pre-deployed on the container platform; Acquire event data generated by the container platform, and convert the current data format of the event data into the data format of a distributed search cluster; The format-converted event data is stored in the distributed search cluster.

3. The method according to claim 2, characterized in that: After storing the format-converted event data in the distributed search cluster, the method further includes: Determine event information corresponding to the event data; the event information includes at least one of event type, event occurrence time, and event level; classifying the event data in the distributed search cluster according to the event information; An index is created for the classified event data according to the event information.

4. The method according to claim 1, characterized in that: The receiving a detection request for the container platform and searching the distributed search cluster for target event data corresponding to the detection request includes: Predetermining detection conditions for the container platform; wherein the detection conditions include a detection period and a detection target, the detection target includes event information to be retrieved, and the event information includes at least one of an event type, an event occurrence time, and an event level; Receiving a detection request for the distributed search cluster generated according to the detection condition; Sending the detection request to the distributed search cluster regularly using the detection period; In response to the detection request being sent to the distributed search cluster, using the detection target to perform data search in the distributed search cluster to determine target event data corresponding to the detection target; The target event data corresponding to the detection target is arranged using a preset arrangement strategy to obtain the target event data corresponding to the detection request.

5. The method according to claim 1, characterized in that: Submitting the target event data, a preset question template and a list of historical abnormal events to a pre-trained language model to obtain an abnormality detection result includes: Obtaining a preset question template and a pre-generated historical abnormal event list; the historical abnormal event list includes abnormal event data detected in a historical detection cycle; Determine the event information of the target event data, and fill the event information into the question template; Packing the target event data, the filled question template and the historical abnormal event list and submitting them to the pre-trained language model; The language model is used to perform anomaly detection on the target event data according to the question template and the historical abnormal event list to obtain an anomaly detection result.

6. The method according to claim 1, characterized in that: The performing exception processing on the container platform according to the exception detection result includes: Determine anomaly detection results of the language model on the target event data; wherein the anomaly detection results include common anomaly data or unknown anomaly data; The exception detection result is processed by using a processing strategy corresponding to the exception detection result to perform exception processing on the container platform.

7. The method according to claim 6, characterized in that: The adopting a processing strategy corresponding to the abnormal detection result to perform abnormal processing on the container platform includes: If the abnormality detection result is determined to be common abnormal data, the container platform is cleared of abnormalities according to the common abnormal data and the historical abnormal event list; If the abnormality detection result is determined to be unknown abnormal data, an abnormality warning prompt of the container platform is generated, and the abnormality warning prompt is sent to the operation and maintenance personnel of the container platform.

8. A data detection device, characterized in that: The device involves a container platform and a distributed search cluster that are in communication connection, and the device includes: A data storage module, used for storing event data obtained from the container platform to the distributed search cluster; A data search module, configured to receive a detection request for the container platform, and search for target event data corresponding to the detection request in the distributed search cluster; An anomaly detection module, used to submit the target event data, a preset question template and a historical abnormal event list to a pre-trained language model to obtain an anomaly detection result; The exception handling module is used to perform exception handling on the container platform according to the exception detection result.

9. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; The memory is used to store computer programs; The processor is used to implement the method according to any one of claims 1 to 7 when executing the program stored in the memory.

10. A computer-readable storage medium having instructions stored thereon, which, when executed by one or more processors, cause the processors to perform the method according to any one of claims 1 to 7.

Images