Big data service background system, device and medium based on distributed search

Abstract

This invention provides a big data service backend system, device, and medium based on distributed search, relating to the field of data processing technology. It includes: a correlation structure construction module for performing full backend service behavior correlation analysis and constructing a service behavior correlation structure within a preset collection period; a correlation edge differentiation module for constructing collaborative behavior substructures and abnormal behavior substructures; a joint analysis module for determining the behavioral characteristics of multiple nodes in the service behavior correlation structure; and a server switching module for automatically switching business requests to a normal big data server when the identification results meet preset routing switching conditions. This solves the technical problem of low accuracy in anomaly detection when identifying abnormal servers in existing technologies, achieving the technical effect of improving the efficiency and reliability of service backend management.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and more specifically to a big data service backend system, device and medium based on distributed search. Background Technology

[0002] With the continuous development of big data service platforms, more and more enterprises are providing data interface services to external entities by building big data service backend systems. Typically, a big data service backend consists of multiple big data servers, deployed in a distributed manner to handle high concurrency access, and provides various big data product services through interface calls. In actual operation, a large number of user requests continuously access the big data service interface, generating a large amount of log data, order data, and access request data across the various data servers. To improve system efficiency, it is usually necessary to manage and analyze this operational data in a unified manner to achieve functions such as log querying, order management, and system operation status monitoring.

[0003] In existing technologies, the operation and management of big data service backends typically involves collecting server logs through a log system, querying the logs using a database or search engine, and storing and statistically analyzing order information through an order database. Furthermore, statistical analysis of access request IP addresses enables basic traffic monitoring and anomaly detection, thereby ensuring system stability to a certain extent.

[0004] However, in real-world large-scale big data service platforms, the systems typically deploy multiple big data servers with complex business call and data interaction relationships between them. Existing technologies, when performing background operational analysis, usually only analyze a single data source (such as log data or order data) independently, lacking the ability to comprehensively analyze the relationships between various operational data. Especially when abnormal orders, abnormal access requests, or abnormal server behavior occur, existing technologies often struggle to identify abnormal correlations between servers at an overall level, resulting in low accuracy in anomaly detection. Summary of the Invention

[0005] This application provides a big data service backend system, equipment, and media based on distributed search, aiming to solve the technical problem of low accuracy in anomaly detection when identifying abnormal servers in existing technologies.

[0006] The first aspect disclosed in this application provides a big data service backend system based on distributed search, the system comprising: The association structure construction module is used to collect log data, order data, and access request data generated by multiple big data servers in the target big data service backend within a preset collection period, perform full backend service behavior association analysis, and construct a service behavior association structure. This service behavior association structure describes the relationship between big data servers, order events, and access requests, including nodes and association edges. The association edge differentiation module is used to separate the association edges in the service behavior association structure, distinguishing between normal collaborative association edges and abnormal difference association edges, and constructing collaborative behavior substructures based on normal collaborative association edges and abnormal behavior substructures based on abnormal difference association edges. The joint analysis module is used to perform joint analysis based on the collaborative behavior substructures and abnormal behavior substructures to determine the behavior characteristics of multiple nodes in the service behavior association structure. The server switching module is used to identify big data server anomalies based on the behavior characteristics of multiple nodes, and automatically switch business requests to the normal big data server when the identification result meets preset routing switching conditions.

[0007] In one possible implementation, within a preset collection period, log data, order data, and access request data generated by multiple big data servers in the target big data service backend are collected. A full backend service behavior correlation analysis is performed to construct a service behavior correlation structure, including: extracting log timestamps, server identifiers, request interface identifiers, and request IP addresses from the log data to obtain a log event processing relationship set; extracting order numbers, product identifiers, order status, and order times from the order data to determine the big data servers involved in the order processing process and obtain an order flow relationship set; extracting the request source IP address, access interface, and access time from the access request data to determine the big data server corresponding to the access request and obtain a request forwarding relationship set; and establishing association edges between the multiple big data servers based on the log event processing relationship set, the order flow relationship set, and the request forwarding relationship set to construct the service behavior correlation structure.

[0008] In one possible implementation, the association edges in the service behavior association structure are subjected to relationship separation processing to distinguish between normal collaborative association edges and abnormal difference association edges. A collaborative behavior substructure is constructed based on the normal collaborative association edges, and an abnormal behavior substructure is constructed based on the abnormal difference association edges. This includes: calculating a behavior anomaly index based on the log status information, order processing status information, and access request status information corresponding to each association edge in the service behavior association structure; when the behavior anomaly index is less than a preset normal threshold, the corresponding association edge is determined to be a normal collaborative behavior association edge; when the behavior anomaly index is greater than the preset normal threshold, the corresponding association edge is determined to be an abnormal behavior association edge.

[0009] In one possible implementation, joint analysis is performed based on the collaborative behavior substructure and the abnormal behavior substructure to determine multiple node behavior characteristics of multiple nodes in the service behavior association structure. This includes: performing a first behavior propagation analysis in the collaborative behavior substructure to aggregate and propagate normal service behavior information to obtain multiple collaborative behavior characteristics of multiple collaborative behavior substructure nodes; performing a second behavior propagation analysis in the abnormal behavior substructure to propagate abnormal difference behavior information to obtain multiple abnormal behavior characteristics of multiple abnormal behavior substructure nodes; and fusing the multiple collaborative behavior characteristics of the multiple collaborative behavior substructure nodes and the multiple abnormal behavior characteristics of the multiple abnormal behavior substructure nodes with the same node according to the service behavior association structure to determine multiple node behavior characteristics.

[0010] In one possible implementation, a first behavior propagation analysis is performed in the collaborative behavior substructure to aggregate and propagate normal service behavior information to obtain multiple collaborative behavior features of multiple collaborative behavior substructure nodes. This includes: obtaining the set of adjacent nodes for each node in the collaborative behavior substructure; performing differential aggregation propagation on the set of adjacent nodes to obtain the differential aggregation propagation result for each node; and fusing the differential aggregation propagation result for each node with the behavior features obtained based on the normal collaborative association edges of the node to obtain multiple node behavior features of multiple nodes.

[0011] In one possible implementation, the method includes: obtaining the behavioral characteristics of adjacent server nodes of the target server node in the abnormal behavior substructure; calculating the behavioral difference information between the target server node and adjacent server nodes; updating the node behavioral characteristics based on the behavioral difference information to generate multiple abnormal behavior characteristics.

[0012] In one possible implementation, when any node in the service behavior association structure exists only in multiple collaborative behavior substructure nodes or multiple abnormal behavior substructure nodes, the behavior feature obtained based on the association edge of any node will be used as the missing collaborative behavior feature or abnormal behavior feature.

[0013] In one possible implementation, the method includes: concatenating the collaborative behavior features and abnormal behavior features corresponding to each node in the service behavior association structure to obtain concatenated features; performing differential calculation on the collaborative behavior features and abnormal behavior features corresponding to each node in the service behavior association structure to obtain differential features; and normalizing the concatenated features and differential features to obtain multiple node behavior features.

[0014] The second aspect of this application discloses an electronic device including a memory and a processor, wherein the memory stores executable instructions, and the processor, when executing the executable instructions stored in the memory, implements any step of the first aspect of this application.

[0015] A third aspect of this application discloses a computer-readable storage medium storing a computer program for performing any step of the first aspect of this application.

[0016] One or more technical solutions provided in this application have at least the following technical effects or advantages: This application employs a relational structure construction module to collect log data, order data, and access request data generated by multiple big data servers in the target big data service backend within a preset collection period. It then performs full-backend service behavior relational analysis to construct a service behavior relational structure. This structure describes the relationships between big data servers, order events, and access requests, including nodes and relational edges. A relational edge differentiation module separates the relational edges in the service behavior relational structure, distinguishing between normal collaborative relational edges and abnormal difference relational edges. It then constructs collaborative behavior substructures based on normal collaborative relational edges and abnormal behavior substructures based on abnormal difference relational edges. A joint analysis module performs joint analysis based on the collaborative behavior substructures and abnormal behavior substructures to determine the behavioral characteristics of multiple nodes within the service behavior relational structure. A server switching module identifies big data server anomalies based on the behavioral characteristics of multiple nodes. When the identification results meet preset routing switching conditions, business requests are automatically switched to the normal big data server. This achieves the technical effect of improving the efficiency of server backend management.

[0017] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, the following are specific embodiments of this application. Attached Figure Description

[0018] Figure 1 This is a schematic diagram of the backend system architecture for a big data service based on distributed search, provided in an embodiment of this application.

[0019] Figure 2 This is a flowchart illustrating the process of constructing a service behavior association structure in a big data service backend system based on distributed search, as provided in an embodiment of this application.

[0020] Figure 3 This is an internal structural diagram of an electronic device provided in an embodiment of this application.

[0021] Explanation of reference numerals in the attached figures: The system includes: a relational structure construction module 11, a relational edge differentiation module 12, a joint analysis module 13, a server switching module 14, a bus 300, a receiver 301, a processor 302, a transmitter 303, a memory 304, and a bus interface 305. Detailed Implementation

[0022] This application provides a big data service backend system based on distributed search, which solves the technical problem of low accuracy in anomaly detection when identifying abnormal servers in the prior art.

[0023] After introducing the basic principles of this application, various non-limiting embodiments of this application will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit this application.

[0024] Example 1, as Figure 1 As shown in the embodiment of this application, a big data service backend system based on distributed search is provided, the system comprising: The association structure construction module 11 is used to collect log data, order data and access request data generated by multiple big data servers in the target big data service backend within a preset collection period, perform full backend service behavior association analysis, and construct a service behavior association structure. The service behavior association structure is used to describe the association relationship between big data servers, order events and access requests, including nodes and association edges. It should be noted that the preset collection period refers to the data collection time window pre-set by the system, such as 1 minute, 5 minutes, or 10 minutes. Service behavior correlation analysis refers to identifying whether there are business correlations between different servers based on the time, source, processing path, and result status of different data. In this solution, nodes preferably represent big data servers, used to carry out subsequent server behavior analysis. Correlation edges represent the business connections between servers formed due to log calls, order flow, or request forwarding, used to reflect the collaborative processing relationship or anomaly transmission relationship between servers.

[0025] For example, within a 5-minute data collection period, server A generates 1000 interface call logs and 300 order processing records, while server B generates 800 interface call logs and 280 order processing records. Simultaneously, a user request is forwarded from server A to server B to complete a product query. At this point, the system can first extract the log timestamp, server identifier, request interface identifier, and request IP address from the log data to identify the log processing relationship of server A calling server B's product query interface. Then, it can extract the order number, product identifier, order status, and order time from the order data to identify the order flow relationship of order O123, which was accepted by server A and processed by server B. Finally, it can extract the request source IP address, access interface, and access time from the access request data to identify the request forwarding relationship of request R456, which first arrives at server A and is then forwarded to server B. Based on these relationships, an association edge can be established between server A and server B, and this process can be repeated for multiple servers in the backend to ultimately form a service behavior association structure.

[0026] By mapping previously isolated log records, order records, and request records to the relationships between servers, the system can no longer only see the local operation of a single server, but also identify the business collaboration links and anomaly propagation links between servers from a full backend perspective. This provides a reliable foundation for subsequent separation of related edge relationships and determination of node behavior characteristics.

[0027] The association edge differentiation module 12 is used to perform relationship separation processing on the association edges in the service behavior association structure, distinguish between normal collaborative association edges and abnormal difference association edges, and construct a collaborative behavior substructure based on the normal collaborative association edges and an abnormal behavior substructure based on the abnormal difference association edges. Specifically, in implementation, for each associated edge in the service behavior association structure, the behavior anomaly index can be calculated by comprehensively considering its corresponding log status information, order processing status information, and access request status information. For example, within a certain collection period, there is an associated edge between server A and server B. The log display interface response time corresponding to this associated edge is 80 milliseconds, the call success rate is 99.8%, the corresponding order status is normal completion, and the corresponding request forwarding status is no timeout and no retries. Then the system can calculate a low behavior anomaly index, such as 0.12. If the preset normal threshold is 0.30, then this associated edge can be determined as a normal collaborative associated edge and included in the collaborative behavior substructure.

[0028] If the logs for another connection edge between server C and server D show continuous timeouts in interface calls, frequent error codes, a high number of request retries, and a large number of failed or suspended orders, a high abnormal behavior index (e.g., 0.78) can be calculated. When this value exceeds a preset normal threshold, the connection edge is identified as an abnormal difference connection edge and included in the abnormal behavior substructure. By performing the above processing on all connection edges in the service behavior connection structure, a collaborative behavior substructure composed of normal business relationships and an abnormal behavior substructure composed of abnormal business relationships can be constructed respectively.

[0029] By separating normal and abnormal business links in the entire backend server relationship, normal and abnormal data are avoided from being mixed together during subsequent behavior propagation analysis. This allows the system to learn the normal collaboration patterns between servers while also being able to identify abnormal propagation characteristics independently, thereby improving the accuracy of subsequent node behavior feature extraction and abnormal server identification.

[0030] The joint analysis module 13 is used to perform joint analysis based on the collaborative behavior substructure and the abnormal behavior substructure to determine the behavior characteristics of multiple nodes in the service behavior association structure. In one embodiment, the collaborative behavior substructure performs normal business behavior propagation analysis on each server node. This includes statistically analyzing metrics such as the server's call frequency in the normal call chain, normal order processing success rate, and the number of normal request forwards. The system also aggregates and calculates the behavioral characteristics of adjacent nodes to obtain the collaborative behavior characteristics of each server node. For example, if server A has normal collaborative relationships with servers B and C within a collection period, the system can count 500 normal interface calls sent by server A to server B and 300 normal order processing requests sent to server C, and obtain the collaborative behavior characteristics of server A through aggregation calculation.

[0031] Subsequently, in the abnormal behavior substructure, abnormal behavior propagation analysis is performed on each server node. This includes metrics such as the number of abnormal orders, the number of abnormal request forwards, and the failure rate of abnormal interface calls, thereby obtaining the abnormal behavior characteristics of each server node. For example, if server D experiences 40 order failures and 25 abnormal interface calls within the same period, corresponding abnormal behavior characteristics can be generated. Finally, based on the service behavior association structure, the collaborative behavior characteristics and abnormal behavior characteristics obtained by the same server node in the two substructures are fused within the same node. This can be achieved through methods such as feature concatenation, differential calculation, or normalization fusion to generate unified node behavior characteristics.

[0032] By unifying the modeling of normal business collaboration behavior and abnormal business difference behavior of the server, the system can characterize the operating status of server nodes from an overall perspective, thereby providing a more comprehensive and accurate behavioral basis for subsequent big data server anomaly identification.

[0033] Server switching module 14 is used to identify big data server anomalies based on the behavioral characteristics of the multiple nodes. When the identification result meets the preset routing switching conditions, the business request is automatically switched to the big data server that is providing normal service.

[0034] The multiple node behavior characteristics output by the joint analysis module 13 are input into the anomaly identification model or judgment rules to calculate the node anomaly probability or anomaly score for each big data server. For example, during a certain collection period, server A's collaborative behavior characteristics show a decrease in normal call links, while its abnormal behavior characteristics show a surge in abnormal orders and an increase in request failure rate. After comprehensive analysis, the node anomaly probability of server A is found to be 0.86. If the system's preset anomaly identification threshold is 0.75, then server A is determined to be an abnormal server. Furthermore, the system can also determine whether server A meets preset routing switching conditions, such as requiring "the node anomaly probability to be greater than 0.75 for two consecutive collection periods, the order failure rate to exceed 15%, and the existence of an available backup server." If server B provides the same big data product service at this time, and its node anomaly probability is only 0.18 and its resource utilization rate is within the normal range, then the system can automatically switch the business requests originally pointing to server A to server B in the routing configuration table.

[0035] Furthermore, such as Figure 2 As shown, within a preset collection period, log data, order data, and access request data generated by multiple big data servers in the target big data service backend are collected. A full backend service behavior correlation analysis is performed to construct a service behavior correlation structure, including: Extract log timestamps, server identifiers, request interface identifiers, and request IP addresses from the log data to obtain a set of log event processing relationships; Extract order number, product identifier, order status, and order time from order data to identify the big data servers involved in the order processing and obtain a set of order flow relationships; Extract the request source IP address, access interface, and access time from the access request data to determine the big data server corresponding to the access request and obtain the request forwarding relationship set. Based on the log event processing relationship set, the order flow relationship set, and the request forwarding relationship set, establish association edges among the multiple big data servers to construct the service behavior association structure.

[0036] In one embodiment, the order number, product identifier, order status, and order time are extracted from the order data. For example, if order O20240618001 is created by server A at 10:02:10, forwarded to server C for order verification at 10:02:12, and returns a "processed successfully" status at 10:02:15, then an order flow relationship exists between server A and server C, and this relationship is included in the order flow relationship set. Next, the request source IP address, access interface, and access time are extracted from the access request data. For example, if the same user request is first received by server A and then forwarded by server A to server B for data query, a request forwarding relationship is formed between server A and server B, and this relationship is included in the request forwarding relationship set. Finally, based on the log event processing relationship set, the order flow relationship set, and the request forwarding relationship set, the system establishes corresponding association edges between server A, server B, and server C, forming a service behavior association structure with multiple big data servers as nodes and inter-server business connections as edges.

[0037] This provides a unified data structure foundation for subsequent edge differentiation, collaborative behavior propagation analysis, abnormal behavior propagation analysis, and node behavior feature determination, enabling the system to analyze business collaboration paths and abnormal propagation paths between big data servers from a full backend perspective, rather than being limited to isolated monitoring of individual servers.

[0038] Furthermore, the association edges in the service behavior association structure are subjected to relationship separation processing to distinguish between normal collaborative association edges and abnormal difference association edges. A collaborative behavior substructure is constructed based on the normal collaborative association edges, and an abnormal behavior substructure is constructed based on the abnormal difference association edges, including: Calculate behavior anomaly indicators based on the log status information, order processing status information, and access request status information corresponding to each associated edge in the service behavior association structure. When the abnormal behavior index is less than the preset normal threshold, the corresponding associated edge is determined to be a normal collaborative behavior associated edge; When the abnormal behavior index is greater than the preset normal threshold, the corresponding associated edge will be determined as an abnormal behavior associated edge.

[0039] It should be noted that log status information is used to characterize the interface response, error code, timeout, and retry status during the call process between the servers corresponding to the associated edge; order processing status information is used to characterize the success, failure, suspension, rollback, and duplicate processing status during the order flow between the servers corresponding to the associated edge; access request status information is used to characterize the success rate, response latency, packet loss, blocking, or abnormal return status during the forwarding of requests between the servers corresponding to the associated edge; preset normal threshold refers to the boundary value set in advance by the system to distinguish between normal collaborative behavior and abnormal difference behavior; normal collaborative associated edge refers to the associated edge whose corresponding business behavior is within the normal fluctuation range and can reflect the normal business collaborative relationship between servers; abnormal behavior associated edge refers to the associated edge whose corresponding business behavior deviates significantly from the normal state and can reflect abnormal call, abnormal order propagation, or abnormal request forwarding relationship.

[0040] Relationship separation processing essentially involves classifying the states of edges in the service behavior association structure, so that subsequent analysis no longer mixes normal and abnormal business relationships, but instead performs targeted analysis in different substructures.

[0041] Furthermore, based on the joint analysis of the collaborative behavior substructure and the abnormal behavior substructure, the behavioral characteristics of multiple nodes in the service behavior association structure are determined, including: The first behavior propagation analysis is performed in the collaborative behavior substructure to aggregate and propagate normal service behavior information, thereby obtaining multiple collaborative behavior features of multiple collaborative behavior substructure nodes. A second behavior propagation analysis is performed in the abnormal behavior substructure to propagate the abnormal difference behavior information and obtain multiple abnormal behavior features of multiple abnormal behavior substructure nodes. Based on the service behavior association structure, multiple collaborative behavior features of multiple collaborative behavior substructure nodes and multiple abnormal behavior features of multiple abnormal behavior substructure nodes are fused together to determine multiple node behavior features.

[0042] Furthermore, a first behavior propagation analysis is performed within the collaborative behavior substructure to aggregate and propagate normal service behavior information, obtaining multiple collaborative behavior features of multiple collaborative behavior substructure nodes, including: Obtain the set of adjacent nodes for each node in the cooperative behavior substructure; Perform differential aggregation propagation on the set of adjacent nodes to obtain the differential aggregation propagation result for each node; The differential aggregation propagation results of each node are fused with the behavioral features obtained from the normal collaborative association edges of the nodes to obtain multiple node behavioral features of multiple nodes.

[0043] In one possible embodiment, a first behavior propagation analysis is performed for each server node in the collaborative behavior substructure. Within a certain collection period, server A's set of adjacent nodes in the collaborative behavior substructure is {server B, server C}, where server A has a normal product query collaborative relationship with server B and a normal order verification collaborative relationship with server C.

[0044] The system first extracts normal behavior information from servers B and C, such as server B's normal interface call success rate of 99.5% and average response time of 70ms, and server C's order processing success rate of 98.8% and average processing latency of 95ms. Then, it extracts the behavior characteristics of server A based on normal collaborative association edges, such as server A's normal request forwarding success rate of 99.2% and average forwarding latency of 22ms. Subsequently, it performs differential aggregation propagation on the set of adjacent nodes, for example, calculating the differences between servers B, C, and A in metrics such as call success rate, processing latency, and order completion rate. These differences are then aggregated with the original normal behavior information of the adjacent nodes to obtain the differential aggregation propagation result for server A. This result is then fused with the behavior characteristics obtained by server A based on normal collaborative association edges to obtain the collaborative behavior characteristics of server A. Similarly, in the abnormal behavior substructure, abnormal difference behavior information such as the number of abnormal order propagations, abnormal request failure rate, and abnormal call timeout ratio caused by abnormal adjacent nodes can be extracted for server A to obtain its abnormal behavior characteristics.

[0045] Furthermore, embodiments of this application also include: Obtain the behavioral characteristics of adjacent server nodes of the target server node in the abnormal behavior substructure; Calculate the behavioral differences between the target server node and its neighboring server nodes; The node behavior characteristics are updated based on the aforementioned behavior difference information, generating multiple abnormal behavior characteristics.

[0046] Furthermore, when any node in the service behavior association structure exists only in multiple collaborative behavior substructure nodes or multiple abnormal behavior substructure nodes, the behavior features obtained from the association edges of any node will be used as the missing collaborative behavior features or abnormal behavior features.

[0047] In one possible embodiment, the set of adjacent nodes of server E in the abnormal behavior substructure is {server F, server G}, where server F has a high order processing failure rate, for example, an order failure rate of 22%, and server G has a high interface call timeout rate, for example, a timeout rate of 15%. The system first extracts the abnormal behavior features corresponding to servers F and G, such as order failure rate, request retries, and interface failure rate. Then, it calculates the behavioral difference information between server E and servers F and G. For example, if server E's order failure rate is 5%, then its difference value with server F is 17%, and its difference value with server G is 10%. Subsequently, the system fuses and updates this behavioral difference information with server E's current abnormal behavior features, for example, by generating new abnormal behavior features through weighted updates or differential propagation, thereby obtaining the abnormal propagation features of server E in the abnormal behavior substructure.

[0048] In actual operation, some server nodes may only participate in the normal business chain and not appear in the abnormal behavior substructure. For example, server H may only exist in the collaborative behavior substructure, and its abnormal behavior characteristics are missing. In this case, the system can generate compensatory abnormal behavior characteristics based on the log call characteristics, order processing success rate, and request forwarding characteristics corresponding to the associated edges of server H to fill in the missing abnormal behavior dimensions. Similarly, for nodes that only exist in the abnormal behavior substructure, compensatory collaborative behavior characteristics can also be generated based on their associated edge behavior information. Through the above mechanism, it can be ensured that each server node in the service behavior association structure ultimately possesses complete collaborative behavior characteristics and abnormal behavior characteristics, thereby providing stable and consistent input data for subsequent node behavior characteristic fusion and big data server anomaly identification.

[0049] Furthermore, embodiments of this application also include: The collaborative behavior features and abnormal behavior features corresponding to each node in the service behavior association structure are concatenated to obtain the concatenated features. Differential calculations are performed on the collaborative behavior features and abnormal behavior features corresponding to each node in the service behavior association structure to obtain differential features; Normalize the spliced ​​features and the difference features to obtain multiple node behavior features.

[0050] It should be noted that feature concatenation refers to combining the collaborative behavior features obtained by the same server node in the collaborative behavior substructure with the abnormal behavior features obtained in the abnormal behavior substructure in a predetermined order, thereby forming a joint feature vector containing information from both types of behavior. Differential calculation refers to performing difference operations on the corresponding dimensions between the collaborative behavior features and the abnormal behavior features, which is used to characterize the degree of behavioral deviation of the same server between normal collaborative mode and abnormal propagation mode.

[0051] In one embodiment, for server A, the collaborative behavior features obtained in the collaborative behavior substructure might include a normal interface call success rate of 0.995%, a normal order completion rate of 0.988%, and an average request forwarding latency of 20ms; the abnormal behavior features obtained in the abnormal behavior substructure might include an abnormal order ratio of 0.03%, an interface error rate of 0.015%, and 5 abnormal request retries. The system first concatenates these two sets of features, and then performs differential calculations on the corresponding dimensions, such as calculating the difference between the normal order completion rate and the abnormal order ratio, and the difference between the normal call success rate and the interface error rate, thereby obtaining a differential feature vector.

[0052] The concatenated features and the differential features are merged, and then normalized (e.g., using min-max normalization or Z-score standardization) is applied to map each feature value to a uniform numerical range, such as between 0 and 1, forming the final node behavior features of server A. The same process is repeated for other nodes in the service behavior association structure, such as server B and server C, to obtain multiple node behavior features.

[0053] By concatenating data to preserve complete information on both normal and abnormal server behaviors, using differential calculations to highlight the degree of abnormal deviations in server behavior, and then normalizing the feature scale to unify the feature scale, the resulting node behavior features contain comprehensive information and have good comparability, thereby significantly improving the accuracy and stability of subsequent big data server anomaly identification.

[0054] Example 2, as Figure 3 The diagram shown is a schematic representation of the structure of an exemplary electronic device of this application. Figure 3 In this document, the bus architecture is represented by bus 300. Bus 300 may include any number of interconnected buses and bridges, and bus 300 connects various circuits including one or more processors represented by processor 302 and memory represented by memory 304. Bus 300 may also connect various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein. Bus interface 305 provides an interface between bus 300 and receiver 301 and transmitter 303. Receiver 301 and transmitter 303 may be the same element, i.e., a transceiver, providing a unit for communicating with various other devices over a transmission medium.

[0055] The memory 304, as a computer-readable storage medium, can be used to store software programs, computer-executable programs, and modules, such as the program instructions / modules corresponding to the distributed search-based big data service backend system in this embodiment. The processor 302 executes various functional applications and data processing of the computer device by running the software programs, instructions, and modules stored in the memory 304, thereby realizing the aforementioned distributed search-based big data service backend system. The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as the combination of these technical features does not contradict each other, it should be considered within the scope of this specification.

[0056] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A big data service back-end system based on distributed search, characterized in that, The system includes: The association structure construction module is used to collect log data, order data and access request data generated by multiple big data servers in the target big data service backend within a preset collection period, perform full backend service behavior association analysis, and construct a service behavior association structure. The service behavior association structure is used to describe the association relationship between big data servers, order events and access requests, including nodes and association edges. The association edge differentiation module is used to perform relationship separation processing on the association edges in the service behavior association structure, distinguish between normal collaborative association edges and abnormal difference association edges, and construct a collaborative behavior substructure based on the normal collaborative association edges and an abnormal behavior substructure based on the abnormal difference association edges. The joint analysis module is used to perform joint analysis based on the collaborative behavior substructure and the abnormal behavior substructure to determine the behavior characteristics of multiple nodes in the service behavior association structure. The server switching module is used to identify big data server anomalies based on the behavioral characteristics of the multiple nodes. When the identification result meets the preset routing switching conditions, the business request is automatically switched to the big data server that is providing normal service.

2. The big data service backend system based on distributed search of claim 1, wherein, Within a preset collection period, log data, order data, and access request data generated by multiple big data servers in the target big data service backend are collected. A full backend service behavior correlation analysis is performed to construct a service behavior correlation structure, including: Extract log timestamps, server identifiers, request interface identifiers, and request IP addresses from the log data to obtain a set of log event processing relationships; Extract order number, product identifier, order status, and order time from order data to identify the big data servers involved in the order processing and obtain a set of order flow relationships; Extract the request source IP address, access interface, and access time from the access request data to determine the big data server corresponding to the access request and obtain the request forwarding relationship set. Based on the log event processing relationship set, the order flow relationship set, and the request forwarding relationship set, establish association edges among the multiple big data servers to construct the service behavior association structure.

3. The big data service backend system based on distributed search of claim 1, wherein, The association edges in the service behavior association structure are subjected to relationship separation processing to distinguish between normal collaborative association edges and abnormal difference association edges. A collaborative behavior substructure is constructed based on the normal collaborative association edges, and an abnormal behavior substructure is constructed based on the abnormal difference association edges, including: Calculate behavior anomaly indicators based on the log status information, order processing status information, and access request status information corresponding to each associated edge in the service behavior association structure. When the abnormal behavior index is less than the preset normal threshold, the corresponding associated edge is determined to be a normal collaborative behavior associated edge; When the abnormal behavior index is greater than the preset normal threshold, the corresponding associated edge will be determined as an abnormal behavior associated edge.

4. The big data service backend system based on distributed search of claim 1, wherein, Based on the joint analysis of the collaborative behavior substructure and the abnormal behavior substructure, the behavioral characteristics of multiple nodes in the service behavior association structure are determined, including: The first behavior propagation analysis is performed in the collaborative behavior substructure to aggregate and propagate normal service behavior information, thereby obtaining multiple collaborative behavior features of multiple collaborative behavior substructure nodes. A second behavior propagation analysis is performed in the abnormal behavior substructure to propagate the abnormal difference behavior information and obtain multiple abnormal behavior features of multiple abnormal behavior substructure nodes. Based on the service behavior association structure, multiple collaborative behavior features of multiple collaborative behavior substructure nodes and multiple abnormal behavior features of multiple abnormal behavior substructure nodes are fused together to determine multiple node behavior features.

5. The big data service backend system based on distributed search of claim 4, wherein, In the collaborative behavior substructure, a first behavior propagation analysis is performed to aggregate and propagate normal service behavior information, obtaining multiple collaborative behavior features of multiple collaborative behavior substructure nodes, including: Obtain the set of adjacent nodes for each node in the cooperative behavior substructure; Perform differential aggregation propagation on the set of adjacent nodes to obtain the differential aggregation propagation result for each node; The differential aggregation propagation results of each node are fused with the behavioral features obtained from the normal collaborative association edges of the nodes to obtain multiple node behavioral features of multiple nodes.

6. The big data services backend system based on distributed search of claim 4, wherein, include: Obtain the behavioral characteristics of the target server node's adjacent server nodes in the abnormal behavior substructure; Calculate the behavioral differences between the target server node and its neighboring server nodes; The node behavior characteristics are updated based on the aforementioned behavior difference information, generating multiple abnormal behavior characteristics.

7. The big data services backend system based on distributed search of claim 4, wherein, When any node in the service behavior association structure exists only in multiple collaborative behavior substructure nodes or multiple abnormal behavior substructure nodes, the behavior feature obtained based on the association edge of any node will be used as the missing collaborative behavior feature or abnormal behavior feature.

8. The big data services backend system based on distributed search of claim 4, wherein, include: The collaborative behavior features and abnormal behavior features corresponding to each node in the service behavior association structure are concatenated to obtain the concatenated features. Differential calculations are performed on the collaborative behavior features and abnormal behavior features corresponding to each node in the service behavior association structure to obtain differential features; Normalize the splicing features and the difference features to obtain multiple node behavior features.

9. An electronic device, comprising: The electronic device includes: Memory, used to store executable instructions; The processor, when executing executable instructions stored in the memory, implements the big data service backend system based on distributed search as described in any one of claims 1-8.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, When the program is executed by the processor, it implements the big data service backend system based on distributed search as described in any one of claims 1-8.

Images