A log vector retrieval system, method and medium based on a cloud native scene

Abstract

A log vector retrieval system, method and medium based on a cloud native scene relate to the technical field of data processing. The log vector server is configured to send a log retrieval request to a corresponding N log vector nodes according to the scheduling information of the received log retrieval request; the N log vector nodes are configured to determine whether the time difference between the storage time of the log retrieval result corresponding to the log retrieval request and the current time is less than a time threshold; if yes, determine K log retrieval results most relevant to the log retrieval request in the data queue; if no, determine K log retrieval results most relevant to the log retrieval request in the data worker and the data queue; and the log vector server is further configured to determine the final retrieval result by integrating the K log retrieval results. Thus, the system uses each log vector node of the k8s node as an independent log data storage and retrieval unit, avoids remote storage dependence, and significantly improves resource utilization.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, in particular to a log vector retrieval system and method based on a cloud native scenario and a medium. BACKGROUND

[0002] With the popularity of cloud native technology, enterprises generally deploy containerized applications using Kubernetes (k8s). Each Pod generates a large amount of log data (such as alarm logs, error logs, etc.) during operation, and these log data are of great value for monitoring application status, troubleshooting, and business analysis.

[0003] Currently, in order to extract useful information from a large amount of log data, traditional solutions rely on centralized storage and retrieval (such as an Elasticsearch cluster), which requires all log data to be sent to a remote server for storage and retrieval. However, the above method cannot fully utilize the local storage resources of k8s nodes, resulting in idle node free storage space and low resource utilization. SUMMARY

[0004] Based on the above problems, the present application provides a log vector retrieval system and method based on a cloud native scenario and a medium, which can avoid remote storage dependency and significantly improve resource utilization.

[0005] The embodiments of the present application disclose the following technical solutions:

[0006] In a first aspect, the present application discloses a log vector retrieval system based on a cloud native scenario, which comprises a log vector server and M log vector nodes deployed on k8s nodes, M being a positive integer, and each log vector node storing vectorized log data; the log vector node stores vectorized log data in the following manner: when a mounted application log change is detected, log data is read; the log data is cut to obtain a plurality of block log data; the plurality of block log data are respectively vectorized, and the vectorized log data is inserted into a data queue data-queue; when the vectorized log data in the data queue data-queue has been maintained for a time length greater than a time length threshold, the vectorized log data is stored in a data worker data-worker.

[0007] The log vector server is configured to receive a log retrieval request, wherein the log retrieval request comprises scheduling information; according to the scheduling information, the log retrieval request is sent to corresponding N log vector nodes, N being a positive integer less than or equal to M;

[0008] The N log vector nodes are configured to, in response to the log retrieval request, determine whether a time difference between a storage time of a log retrieval result corresponding to the log retrieval request and a current time is less than a time threshold; if yes, determine, in the data queue data-queue, K log retrieval results most relevant to the log retrieval request; if no, determine, in the data worker data-worker and the data queue data-queue, K log retrieval results most relevant to the log retrieval request, K being a positive integer; and send the K log retrieval results to the log vector server.

[0009] The log vector server is further configured to determine a final retrieval result by integrating the K log retrieval results from the N log vector nodes.

[0010] Optionally, the N log vector nodes are specifically configured to determine the K log retrieval results most relevant to the log retrieval request by using a hierarchical navigable small world graph (HNSW) algorithm.

[0011] Optionally, the log vector server is specifically configured to:

[0012] The final retrieval result is determined by determining, in the K log retrieval results of the N log vector nodes, J log retrieval results most relevant to the log retrieval request, J being a positive integer less than N×K.

[0013] Optionally, the reading of the log data when detecting a change in the mounted application log comprises:

[0014] The log data is read from a historical reading position when detecting a change in the mounted application log.

[0015] Optionally, the cutting of the log data to obtain a plurality of piecewise log data comprises:

[0016] The log data is cut according to a log size or a log reading time to obtain a plurality of piecewise log data.

[0017] Optionally, the cutting of the log data to obtain a plurality of piecewise log data comprises:

[0018] The log data is cut in three levels of collection, partition and segment to obtain collection log data, each of the collection log data comprising a plurality of partition log data, and each of the partition log data comprising a plurality of piecewise log data.

[0019] In a second aspect, the application discloses a log vector retrieval method based on a cloud native scene, which is applied to a log vector retrieval system based on a cloud native scene, the system comprising a log vector server and M log vector nodes deployed on k8s nodes, M being a positive integer, and each of the log vector nodes storing vectorized log data; the log vector nodes store the vectorized log data in the following manner: when detecting a change in a mounted application log, reading log data; cutting the log data to obtain a plurality of block log data; vectorizing the plurality of block log data respectively, and inserting the vectorized log data into a data queue data-queue; when the vectorized log data is kept in the data queue data-queue for a time length greater than a time length threshold, storing the vectorized log data into a data worker data-worker;

[0020] The method comprises:

[0021] The log vector server receives a log retrieval request, the log retrieval request comprising scheduling information; according to the scheduling information, the log retrieval request is sent to corresponding N log vector nodes, N being a positive integer less than or equal to M;

[0022] The N log vector nodes determine, in response to the log retrieval request, whether a time difference between a storage time of a log retrieval result corresponding to the log retrieval request and a current time is less than a time threshold; if yes, K log retrieval results most relevant to the log retrieval request are determined in the data queue data-queue; if no, K log retrieval results most relevant to the log retrieval request are determined in the data worker data-worker and the data queue data-queue, K being a positive integer; the K log retrieval results are sent to the log vector server;

[0023] The log vector server determines a final retrieval result by integrating the K log retrieval results from the N log vector nodes.

[0024] Optionally, determining the K log retrieval results most relevant to the log retrieval request comprises:

[0025] The K log retrieval results most relevant to the log retrieval request are determined by a hierarchical navigable small world graph HNSW algorithm.

[0026] Optionally, determining the final retrieval result by integrating the K log retrieval results from the N log vector nodes comprises:

[0027] The final retrieval result is determined by determining J log retrieval results most relevant to the log retrieval request from the K log retrieval results of the N log vector nodes, J being a positive integer less than N×K.

[0028] Optionally, the reading log data when detecting the change of the mounted application log comprises:

[0029] The log data is read from the historical reading position when detecting the change of the mounted application log.

[0030] Optionally, the cutting the log data to obtain a plurality of piecewise log data comprises:

[0031] The log data is cut according to the log size or the log reading time to obtain a plurality of piecewise log data.

[0032] Optionally, the cutting the log data to obtain a plurality of piecewise log data comprises:

[0033] The log data is cut in three levels of collection, partition and segment to obtain collection log data, each of the collection log data comprising a plurality of partition log data, and each of the partition log data comprising a plurality of piecewise log data.

[0034] In a third aspect, the present application discloses a computer readable medium having a computer program stored thereon, the computer program being executed by a processor to implement each step of the log vector retrieval method based on a cloud native scenario as described in the second aspect.

[0035] Compared with the prior art, the present application has the following beneficial effects:

[0036] The embodiment of the application discloses a log vector retrieval system and method based on a cloud native scene and a medium, the system comprises: a log vector server and M log vector nodes deployed on k8s nodes, M is a positive integer, and vectorized log data is stored in each log vector node; the log vector node stores the vectorized log data in the following manner: when detecting that the mounted application log changes, the log data is read; the log data is cut to obtain a plurality of block log data; the vectorized log data is inserted into a data queue data-queue by being respectively vectorized; when the vectorized log data is stored in the data worker data-worker for a time length greater than a time length threshold in the data queue data-queue; the log vector server is used for receiving a log retrieval request, the log retrieval request comprising scheduling information; according to the scheduling information, the log retrieval request is sent to the corresponding N log vector nodes, N is a positive integer less than or equal to M; the N log vector nodes are used for determining whether the time difference between the storage time of the log retrieval result corresponding to the log retrieval request and the current time is less than the time threshold in response to the log retrieval request; if yes, the K log retrieval results most relevant to the log retrieval request are determined in the data queue data-queue; if not, the K log retrieval results most relevant to the log retrieval request are determined in the data worker data-worker and the data queue data-queue, K is a positive integer; the K log retrieval results are sent to the log vector server; the log vector server is also used for determining the final retrieval result by integrating the K log retrieval results from the N log vector nodes. Therefore, the log vector retrieval system based on the cloud native scene provided by the embodiment of the application uses each log vector node of the k8s node as an independent log data storage and retrieval unit, avoids remote storage dependence, and significantly improves the resource utilization. BRIEF DESCRIPTION OF DRAWINGS

[0037] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiment or prior art description. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0038] Figure 1 A schematic diagram of a log vector retrieval system based on a cloud native scene provided by the embodiment of the present application;

[0039] Figure 2 A schematic diagram of a log vector node storing vectorized log data provided by the embodiment of the present application;

[0040] Figure 3 A schematic diagram of a three-level cutting provided for an embodiment of the present application;

[0041] Figure 4 A schematic diagram of obtaining a plurality of chunked log data provided for an embodiment of the present application;

[0042] Figure 5 A schematic diagram of querying log retrieval results provided for an embodiment of the present application;

[0043] Figure 6 A schematic diagram of an HNSW algorithm provided for an embodiment of the present application;

[0044] Figure 7 A schematic diagram of integrating log retrieval results provided for an embodiment of the present application;

[0045] Figure 8 A schematic diagram of another integrating log retrieval results provided for an embodiment of the present application;

[0046] Figure 9 A schematic diagram of a log vector server performing log vector retrieval for a plurality of log retrieval requests provided for an embodiment of the present application;

[0047] Figure 10 A flowchart of a log vector retrieval method based on a cloud-native scenario provided for an embodiment of the present application. DETAILED DESCRIPTION

[0048] As described above, in order to extract useful information from massive log data, the traditional scheme relies on centralized storage and retrieval (such as Elasticsearch cluster), and all log data needs to be sent to a remote server for storage and retrieval. However, the above method cannot fully utilize the local storage resources of k8s nodes, resulting in idle node free storage space and low resource utilization.

[0049] The inventor has proposed a log vector retrieval system, method and medium based on a cloud-native scenario, which uses each log vector node of the k8s node as an independent log data storage and retrieval unit, without the need for additional external storage servers, avoiding remote storage dependence, and significantly improving resource utilization. Moreover, each log vector node returns a local Top-K result instead of full data, reducing network transmission and aggregation calculation overhead, and the log vector server performs secondary screening on N x K log retrieval results to select a local Top-J result, improving the accuracy of log vector retrieval.

[0050] In the following, the technical solutions in the embodiments of the present application will be described clearly and completely in conjunction with the drawings in the embodiments of the present application, so that those skilled in the art can better understand the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.

[0051] Referring to Figure 1 The figure is a schematic diagram of a log vector retrieval system based on a cloud native scene provided by an embodiment of the present application. The log vector retrieval system based on the cloud native scene comprises a log vector server (log-Vector-Server) and M log vector nodes (log-vector-node) deployed on k8s nodes, M being a positive integer.

[0052] S101: The log vector server is configured to receive a log retrieval request, the log retrieval request comprising scheduling information.

[0053] When it is necessary to trigger troubleshooting, performance analysis, audit compliance, etc., the log retrieval request is initiated by a user (such as an operation and maintenance personnel) through a CLI / Web interface and sent to the log vector server. The scheduling information needs to be carried in the log retrieval request. The scheduling information comprises an application name (i.e. the identification of a containerized application to be retrieved, such as app-a), a Pod identification (i.e. the unique ID of a target Pod, such as pod-123), and a K8s node label (such as {"zone":"us-east-1"}), etc.

[0054] It can be understood that the log vector server can determine the range of log vector nodes (i.e. the corresponding N log vector nodes) to which the log retrieval request needs to be distributed by analyzing the scheduling information in the log retrieval request, avoiding full-cluster broadcasting, thereby reducing network bandwidth occupation and computing resource consumption.

[0055] S102: The log vector server is configured to send the log retrieval request to the corresponding N log vector nodes according to the scheduling information, N being a positive integer less than or equal to M.

[0056] For example, if the scheduling information of the log retrieval request indicates that the error log of application A is requested to be queried, and the Pods of application A are distributed on log vector node 1 and log vector node 2, then the log retrieval request will only be sent to log vector node 1 and log vector node 2.

[0057] It can be understood that when the application is scaled up (new Pods are added), the scheduling information will automatically cover the new log vector nodes, without manual intervention.

[0058] S103: The N log vector nodes are configured to, in response to the log retrieval request, determine whether a time difference between a storage time of the log retrieval result corresponding to the log retrieval request and a current time is less than a time threshold; if yes, determine, in the data queue data-queue, the K log retrieval results most relevant to the log retrieval request; if no, determine, in the data worker data-worker and the data queue data-queue, the K log retrieval results most relevant to the log retrieval request, K being a positive integer.

[0059] The log vector node (log-vector-node) has a log data storage capability, and each log vector node stores vectorized log data. Referring to Figure 2 The figure is a schematic diagram of a log vector node storing vectorized log data according to an embodiment of the present application. The log vector node stores vectorized log data in the following manner:

[0060] A1: When detecting a change in the mounted application log, read the log data.

[0061] When a user mounts an application APP-Log on a log vector node through a console or a kubectl command line, the log vector node continuously listens to whether a change occurs in the mounted application log. If yes, read the log data.

[0062] In a specific implementation, if the log data is read for the first time, the log file is read in full to obtain the log data. If the log data is not read for the first time, the log file can be read incrementally from the historical reading position to obtain the log data (i.e., only the changed part is read), thereby avoiding repeated calculation.

[0063] A2: Cut the log data to obtain a plurality of chunked log data.

[0064] In a specific implementation, the log data can be cut according to the log size or the log reading time to obtain a plurality of chunked log data.

[0065] In another specific implementation, the log data can also be cut in multiple stages to achieve efficient management and retrieval. Referring to Figure 3 The figure is a schematic diagram of three-stage cutting according to an embodiment of the present application. As Figure 3As shown, the cutting adopts a three-level hierarchical structure: the first level, Collection of log data: each application service is taken as an independent Collection, and for example, the log data of an order service (order-service) constitutes a Collection. The second level, Partition of log data: each Collection is divided into multiple Partitions according to a time window (such as every hour or every day), and for example, order-service-20240501 represents the log data partition of the application on May 1, 2024. The third level, Segment of log data: the log data in each Partition is further cut into multiple Segments according to size (such as every 100 MB) or a fixed number of entries, and for example, a Partition can contain segment-0001 to segment-0010. Thus, this hierarchical design enables parallel processing and precise positioning of log data when stored and retrieved. Figure 3 It can be known that each Collection of log data includes multiple Partitions of log data, and each Partition of log data includes multiple Segments of log data.

[0066] A3: By respectively performing vectorization processing on the multiple Segments of log data, the vectorized log data is inserted into a data queue data-queue.

[0067] In a specific implementation manner, the multiple Segments of log data can be first respectively subjected to structured processing, and key fields are extracted, such as timestamp information (used for time range filtering), log level information (used for priority sorting), message content information, and the like. Subsequently, a pre-trained text embedding model (such as BERT, Sentence-BERT) is used to perform vectorization processing on the structured log result, and the vectorized log data is inserted into the data queue data-queue. The data queue data-queue can be understood as queue buffer data, and can solve the I / O bottleneck caused by directly writing high-frequency small data into a disk (for example, when thousands of logs are written per second, each log is only a few KB, but the disk addressing overhead is large).

[0068] It can be understood that in the process of inserting the vectorized log data into the data queue data-queue, a Segment of log data is written first, and when the capacity of the Segment of log data exceeds the limit, a new Segment of log data is allocated for continuous insertion.

[0069] A4: When the vectorized log data in the data queue data-queue is maintained for a time length greater than a time length threshold, the vectorized log data is stored into a data worker data-worker.

[0070] The time length threshold represents the maximum allowed residence time length of the log data in the data queue data-queue. For example, the time length threshold can be 5 minutes.

[0071] It should be noted that the time length threshold can be dynamically adjusted according to the node memory usage, for example, when the node memory usage exceeds 80%, the time length threshold is automatically shortened (for example, from 5 minutes to 3 minutes) to speed up queue cleaning.

[0072] Referring to Figure 4 , the figure is a schematic diagram provided by an embodiment of the present application for obtaining a plurality of segmented log data. In another specific implementation, when the ratio of the amount of vectorized log data stored in the data queue data-queue to the total capacity of the queue is greater than the storage amount threshold, the first vectorized log data (i.e. the queue head data block) in the data queue data-queue is automatically stored into the data worker data-worker for persistent storage. For example, the storage amount threshold can be 80%. Thus, memory overflow can be avoided by space dimension control, while optimizing the batch processing efficiency of disk I / O, which is suitable for cloud native scenarios with large fluctuations in log generation rate.

[0073] It should be noted that the storage amount threshold can be dynamically adjusted according to the historical log generation rate, for example, when the historical 1-hour log generation rate is higher than the rate threshold (such as the peak rate within 1 hour exceeding 100,000 pieces / minute), the storage amount threshold is automatically reduced (for example, from 80% to 70%) to speed up queue cleaning.

[0074] It can be understood that after the vectorized log data is stored into the data worker data-worker, a unique index can also be assigned to each vectorized log data, thereby improving the retrieval efficiency. For example, the index can be as follows: order-service-20240501-segment-0001-00001234. And the memory occupation in the data queue data-queue can also be released to avoid memory overflow.

[0075] The above paragraph explains the log data storage capacity of the log vector node, and the following explains the vector retrieval capability of the log vector node:

[0076] Referring to Figure 5 , the figure is a schematic diagram for querying log retrieval results provided by an embodiment of the present application. Then, the log vector node can dynamically select the retrieval source according to the storage time of the log data after receiving the log retrieval request.

[0077] Specifically, the log vector node needs to determine whether the time difference Δt between the storage time of the log retrieval result corresponding to the log retrieval request and the current time is less than a time threshold (e.g., 5 minutes).

[0078] If yes, only in the data queue data-queue, the K log retrieval results most relevant to the log retrieval request are determined (hot data, reducing retrieval delay). In this way, the persistent data in the data worker data-worker does not need to be loaded, thereby improving query efficiency. In a specific implementation manner, the K log retrieval results most relevant to the log retrieval request can be determined by a Hierarchical Navigable Small World Graph (HNSW) algorithm. See Figure 6 The figure is a schematic diagram of an HNSW algorithm provided by an embodiment of the present application. HNSW is a graph-based indexing algorithm, and the feature is to construct a hierarchical graph structure. The bottom layer includes nodes of all log data, the higher the layer, the more the number of nodes of log data decreases layer by layer, and the distance between nodes is larger, so that similar nodes can be quickly approached during query, and the retrieval performance and recall effect are very high.

[0079] If no, in the data worker data-worker and the data queue data-queue, the K log retrieval results most relevant to the log retrieval request are determined (hot data and cold data, both reducing retrieval delay and improving retrieval recall rate). For example, the K log retrieval results most relevant to the log retrieval request can be respectively determined in the data worker data-worker and the data queue data-queue (i.e., 2K log retrieval results are determined), and then sorted again according to the relevance score, and the K log retrieval results with the highest relevance score are taken as the final K log retrieval results.

[0080] S104: The N log vector nodes are configured to send the K log retrieval results to the log vector server.

[0081] The log vector node transmits the K log retrieval results to the log vector server in a lightweight format.

[0082] It can be understood that the log vector node can combine the K log retrieval results into a single data packet for sending, or can send the K log retrieval results in K data packets. For this, the present application does not make any limitation.

[0083] S105: The log vector server is configured to determine the final retrieval result by integrating the K log retrieval results from the N log vector nodes.

[0084] The query aggregation layer of the log vector server collects K log retrieval results of N log vector nodes, that is, a total of N x K log retrieval results.

[0085] Referring to Figure 7 FIG. 4 is a schematic diagram of integrating log retrieval results according to an embodiment of the present application. In a specific implementation, the N x K log retrieval results can be integrated to obtain N x K final retrieval results.

[0086] Referring to Figure 8 FIG. 5 is another schematic diagram of integrating log retrieval results according to an embodiment of the present application. In another specific implementation, the N x K log retrieval results can be further filtered, and J log retrieval results most relevant to the log retrieval request are determined from the N x K log retrieval results, and the final retrieval results are determined, J being a positive integer less than N x K.

[0087] It can be understood that the above embodiments are embodiments of the log vector server performing log vector retrieval for one log retrieval request. In actual application, the log vector server can also perform log vector retrieval for multiple log retrieval requests at the same time, as shown in Figure 9 . Figure 9 FIG. 6 is a schematic diagram of a log vector server performing log vector retrieval for multiple log retrieval requests according to an embodiment of the present application.

[0088] In summary, the present application discloses a log vector retrieval system based on a cloud native scenario. Each log vector node of a k8s node is used as an independent log data storage and retrieval unit, and no additional external storage server is needed, avoiding remote storage dependence and significantly improving resource utilization. Moreover, each log vector node returns a local Top-K result instead of full data, reducing network transmission and aggregation calculation overhead. Furthermore, the log vector server performs secondary filtering on N x K log retrieval results to select a local Top-J result, improving the accuracy of log vector retrieval.

[0089] Referring to Figure 10Fig. 1 is a flowchart of a log vector retrieval method based on a cloud-native scenario according to an embodiment of the present application. The method is applied to a log vector retrieval system based on a cloud-native scenario, which comprises a log vector server and M log vector nodes deployed on k8s nodes, M being a positive integer, and each of the log vector nodes storing vectorized log data. The log vector nodes store vectorized log data in the following manner: when a change in the mounted application log is detected, log data is read; the log data is cut to obtain a plurality of chunked log data; the plurality of chunked log data is respectively vectorized to insert the vectorized log data into a data queue data-queue; and when the vectorized log data is kept in the data queue data-queue for a time length greater than a time threshold, the vectorized log data is stored in a data worker data-worker.

[0090] The method comprises:

[0091] S1101: The log vector server receives a log retrieval request, the log retrieval request comprising scheduling information; and according to the scheduling information, the log retrieval request is sent to the corresponding N log vector nodes, N being a positive integer less than or equal to M.

[0092] S1102: The N log vector nodes determine whether the time difference between the storage time of the log retrieval result corresponding to the log retrieval request and the current time is less than a time threshold in response to the log retrieval request; if yes, the K log retrieval results most relevant to the log retrieval request are determined in the data queue data-queue; and if no, the K log retrieval results most relevant to the log retrieval request are determined in the data worker data-worker and the data queue data-queue, K being a positive integer; and the K log retrieval results are sent to the log vector server.

[0093] S1103: The log vector server determines the final retrieval result by integrating the K log retrieval results from the N log vector nodes.

[0094] In a specific implementation, determining the K log retrieval results most relevant to the log retrieval request comprises:

[0095] The K log retrieval results most relevant to the log retrieval request are determined by a hierarchical navigable small world graph HNSW algorithm.

[0096] In a specific implementation, the final retrieval result is determined by integrating the K log retrieval results from the N log vector nodes, comprising:

[0097] Determine the final retrieval result by determining J log retrieval results most relevant to the log retrieval request from K log retrieval results of N log vector nodes, J being a positive integer less than N*K.

[0098] In a specific implementation, when a change in the mounted application log is detected, the log data is read, including:

[0099] When a change in the mounted application log is detected, the log data is read from the historical reading position.

[0100] In a specific implementation, the log data is segmented to obtain a plurality of segmented log data, including:

[0101] The log data is segmented according to the log size or the log reading time to obtain a plurality of segmented log data.

[0102] In a specific implementation, the log data is segmented to obtain a plurality of segmented log data, including:

[0103] The log data is segmented in three levels of collection, partition and segment to obtain collection log data, each collection log data including a plurality of partition log data, and each partition log data including a plurality of segmented log data.

[0104] To sum up, the application discloses a log vector retrieval method based on a cloud native scenario, which uses each log vector node of a k8s node as an independent log data storage and retrieval unit, does not need an additional external storage server, avoids remote storage dependence, and significantly improves resource utilization. Moreover, each log vector node returns a local Top-K result instead of full data, reducing network transmission and aggregation calculation overhead, and the log vector server performs secondary screening on N*K log retrieval results to select a local Top-J result, improving the accuracy of log vector retrieval.

[0105] The application discloses a computer readable medium having a computer program stored thereon, the computer program being executed by a processor to implement each step of the log vector retrieval method based on the cloud native scenario as described in the second aspect, and has the beneficial effects of the log vector retrieval method based on the cloud native scenario as described in the second aspect.

[0106] It should be noted that each of the embodiments described in the specification of the present application adopts a progressive mode for description, and the same or similar parts between the embodiments can be mutually referred to. Each of the embodiments focuses on the differences from other embodiments. In particular, the device and system embodiments are described more simply because they are basically similar to the method embodiments, and the relevant parts can be referred to the part of the method embodiments. The above-described device and system embodiments are only illustrative, and the units described as separate components can or can not be physically separated, and the components indicated as units can or can not be physical units, i.e., they can be located in one place or distributed on multiple network units. Part or all of the modules can be selected to achieve the purpose of the embodiments according to actual needs. Those skilled in the art can understand and implement it without creative labor.

[0107] The above describes only one specific embodiment of the present application, but the protection scope of the present application is not limited to this. Any skilled person in the art can easily think of changes or replacements within the technical range disclosed in the present application, which should be covered in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A log vector retrieval system based on a cloud-native scenario, characterized in that, The system comprises a log vector server and M log vector nodes deployed on k8s nodes, M being a positive integer, and each of the log vector nodes storing vectorized log data; The log vector node stores vectorized log data in the following manner: when a mounted application log change is detected, log data is read; the log data is subjected to three-level segmentation of "collection-partition-segment", to obtain collection log data, each of which comprises multiple partition log data, and each of the partition log data comprises multiple segment log data; the multiple segment log data are subjected to vectorization processing respectively, and the vectorized log data are inserted into a data queue data-queue; when the vectorized log data remain in the data queue data-queue for a time length greater than a time length threshold, the vectorized log data are stored into a data worker data-worker; The log vector server is configured to receive a log retrieval request, the log retrieval request comprising scheduling information; and send the log retrieval request to corresponding N log vector nodes according to the scheduling information, N being a positive integer less than or equal to M; The N log vector nodes are configured to, in response to the log retrieval request, determine whether a time difference between a storage time of a log retrieval result corresponding to the log retrieval request and a current time is less than a time threshold; if yes, determine, in the data queue data-queue, K log retrieval results most relevant to the log retrieval request by a hierarchical navigable small world graph HNSW algorithm; if not, determine, in the data worker data-worker and the data queue data-queue, K log retrieval results most relevant to the log retrieval request by the hierarchical navigable small world graph HNSW algorithm, K being a positive integer; and send the K log retrieval results to the log vector server; The log vector server is further configured to determine, in the K log retrieval results of the N log vector nodes, J log retrieval results most relevant to the log retrieval request as final retrieval results, J being a positive integer less than N×K.

2. The system of claim 1, wherein, The reading of log data when a mounted application log change is detected comprises: When a mounted application log change is detected, reading log data from a historical reading position.

3. The system of claim 1, wherein, The segmentation of the log data to obtain multiple segment log data comprises: Segmenting the log data according to log size or log reading time to obtain multiple segment log data.

4. A log vector retrieval method based on a cloud-native scenario, characterized in that, The application discloses a log vector retrieval system applied to a cloud native scene, which comprises a log vector server and M log vector nodes deployed on k8s nodes, M is a positive integer, and each log vector node stores vectorized log data; the log vector node stores the vectorized log data in the following manner: when detecting that the mounted application log changes, reading log data; cutting the log data to obtain a plurality of block log data; through vectorization processing on the plurality of block log data, the vectorized log data is inserted into a data queue data-queue; when the vectorized log data is stored in the data worker data-worker when the time length of the vectorized log data in the data queue data-queue is greater than the time length threshold; the method comprises the following steps: The method comprises: The log vector server receives a log retrieval request, and the log retrieval request comprises scheduling information; according to the scheduling information, the log retrieval request is sent to corresponding N log vector nodes, N is a positive integer less than or equal to M; The N log vector nodes respond to the log retrieval request, determine whether the time difference between the storage time of the log retrieval result corresponding to the log retrieval request and the current time is less than the time threshold; if yes, the K log retrieval results most relevant to the log retrieval request are determined in the data queue data-queue through a hierarchical navigable small world graph HNSW algorithm; if not, the K log retrieval results most relevant to the log retrieval request are determined in the data worker data-worker and the data queue data-queue through a hierarchical navigable small world graph HNSW algorithm, K is a positive integer; the K log retrieval results are sent to the log vector server; The log vector server determines J log retrieval results most relevant to the log retrieval request as the final retrieval result in the K log retrieval results of the N log vector nodes, J is a positive integer less than N x K. The computer program is executed by the processor to realize each step of the log vector retrieval method based on the cloud native scene.

5. A computer readable medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to realize each step of the log vector retrieval method based on the cloud native scene.

Images