Link log data processing method and device, equipment and medium
By determining log object information in the microservice architecture system and associating service log data, the problem of inability to effectively associate and store request logs in the microservice architecture system in the prior art is solved, and efficient fault location and troubleshooting is achieved.
Patent Information
- Application Number
- CN202411623408.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-14
- Publication Date
- 2025-02-18
AI Technical Summary
In microservice architecture systems, it is difficult for the prior art to associate all service log data involved in a request in a sequence of the entire steps, resulting in the exceptions that occur in the request cannot be located which service unit is causing the problem. The analysis and troubleshooting problem is more efficient Low.
By determining the log object information corresponding to the target service request, including unique link identification, subject identification information, request method containing annotation information, service unit identification information involved in the request, and request information with each service unit, based on these information Each service unit makes a service request and determines the target link log data when the service unit completes the corresponding service. Then, the target link log data is stored in the target distributed system through the message queue.
It realizes that all service log data involved in a request are associated in a sequence of the entire steps, and can efficiently locate fault links based on the link log, improving service troubleshooting efficiency.
Smart Images
Figure CN119484257A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of data processing technology, and in particular to a link log data processing method, device, equipment and medium. Background Art
[0002] With the deepening of digital transformation, microservice architecture has become the mainstream architecture model of enterprises. In the microservice architecture model, it is necessary to track the occurrence of several or even dozens of service units. To be familiar with the operating status of the system, log recording and monitoring are crucial. At present, the log data processing method applied to the microservice architecture system is: print the business logs and operation logs of all service units through the collection console.
[0003] However, the above log processing method cannot associate all service log data involved in a request according to the sequence of the entire steps when a single client request involves multiple service units. As a result, it is impossible to locate which service unit caused the problem when an exception occurs in the request, and the efficiency of analyzing and troubleshooting the problem is low. Summary of the invention
[0004] The present invention provides a link log data processing method, device, equipment and medium, so as to efficiently locate the fault link according to the link log, thereby improving the service fault troubleshooting efficiency.
[0005] In a first aspect, an embodiment of the present invention provides a link log data processing method, the method comprising:
[0006] When a target service request is received, determining log object information corresponding to the target service request; wherein the log object information includes a unique link identifier, subject identifier information, a request method including annotation information, identifier information of at least one service unit involved in the request, and request information corresponding to each of the service units;
[0007] Performing a service request on each of the service units based on the log object information, and determining target link log data corresponding to the target service request when each of the service units completes a corresponding service;
[0008] In the case where the annotation information included in the request method meets a preset annotation condition, the target link log data is stored in a target distributed system by way of a message queue based on the subject identification information.
[0009] In a second aspect, an embodiment of the present invention further provides a link log data processing device, the device comprising:
[0010] A log object determination module, configured to determine log object information corresponding to a target business request when receiving the target business request; wherein the log object information includes a unique link identifier, subject identifier information, a request method including annotation information, identifier information of at least one service unit involved in the request, and request information corresponding to each of the service units;
[0011] A link log determination module, configured to make a service request to each of the service units based on the log object information, and determine target link log data corresponding to the target service request when each of the service units completes the corresponding service;
[0012] The link log storage module is used to store the target link log data in the target distributed system through a message queue based on the subject identification information when the annotation information contained in the request method meets the preset annotation condition.
[0013] In a third aspect, an embodiment of the present invention further provides an electronic device, the electronic device comprising:
[0014] one or more processors;
[0015] a storage device for storing one or more programs,
[0016] When one or more programs are executed by one or more processors, the one or more processors implement a link log data processing method as described in any one of the embodiments of the present invention.
[0017] In a fourth aspect, an embodiment of the present invention further provides a storage medium comprising computer executable instructions, which, when executed by a computer processor, are used to execute a link log data processing method as described in any one of the embodiments of the present invention.
[0018] The technical solution of the embodiment of the present invention is to determine the log object information corresponding to the target business request when receiving the target business request, wherein the log object information includes a unique link identifier, subject identifier information, a request method containing annotation information, identifier information of at least one service unit involved in the request, and request information corresponding to each service unit, and then, based on the log object information, a service request is made to each service unit, and when each service unit completes the corresponding service, the target link log data corresponding to the target business request is determined, so that, when the annotation information contained in the request method meets the preset annotation condition, the target link log data is stored in the target distributed system by means of a message queue based on the subject identifier information. The technical solution of this embodiment can generate link log data through a unique link identifier, and can associate all service log data involved in a request according to the sequence of the entire steps, and can efficiently locate the fault link according to the link log, thereby improving the efficiency of service fault troubleshooting. BRIEF DESCRIPTION OF THE DRAWINGS
[0019] In order to more clearly illustrate the technical solutions of the exemplary embodiments of the present invention, the following briefly introduces the drawings required to describe the embodiments. Obviously, the drawings introduced are only drawings of a part of the embodiments to be described in the present invention, rather than all the drawings. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0020] Figure 1 A flowchart of a link log data processing method provided by an embodiment of the present invention;
[0021] Figure 2 A flowchart of another link log data processing method provided by an embodiment of the present invention;
[0022] Figure 3 A technical schematic diagram of a method for processing link log data according to an embodiment of the present invention;
[0023] Figure 4 A schematic diagram of the structure of a link log data processing device provided by an embodiment of the present invention;
[0024] Figure 5 The present invention is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0025] Figure 1A flow chart of a link log data processing method provided in an embodiment of the present invention. This embodiment is applicable to situations where log data in a microservice architecture system needs to be processed. The method can be executed by a link log data processing device, which can be implemented in the form of software and / or hardware. The hardware can be an electronic device, such as a mobile terminal, a PC or a server.
[0026] The technical solution provided in this embodiment can be applied to any microservice architecture system. Multiple service units that can implement different service functions are deployed in the microservice architecture system. The service unit is the core of the microservice architecture. Each service unit is an independent, autonomous, reusable software module that usually implements a specific business function. Each service unit has its own code, data, and resources, and can be deployed, expanded, and upgraded independently. Service units interact and collaborate through lightweight communication mechanisms. In a microservice architecture system, a link refers to a set of request paths between different service units required to complete a business request. For example, when a client initiates a business request, this request requires multiple service units to perform multiple services, such as service unit A, service unit B, and service unit C. The combined path of these service units is a link.
[0027] like Figure 1 As shown, the link log data processing method includes:
[0028] S110 . When a target service request is received, log object information corresponding to the target service request is determined.
[0029] In this embodiment, the microservice architecture system can respond to various types of business requests from different clients. The target business request refers to the business request currently received from a client during the operation of the microservice architecture system, which is intended to achieve a specific business goal or requirement.
[0030] The log object information refers to the log information related to the target service request recorded through the log object. Specifically, the log object information includes a unique link identifier, subject identifier information, a request method including annotation information, identifier information of at least one service unit involved in the request, and request information corresponding to each service unit.
[0031] Next, the specific data content included in the log object information can be described in detail.
[0032] A unique link identifier (TraceId for short) is a unique identifier that can be assigned to each request in order to track the calling process of a business request between multiple service units. This identifier runs through the entire request life cycle, ensuring that the source, path, and final processing result of each request can be accurately located and tracked in the distributed system. The unique link identifier is used to associate the logs of different service units. Specifically, on each service unit, by passing the unique link identifier, all logs related to the target business request can be associated. In this way, even on different service units, the complete request log can be retrieved through the unique link identifier, which facilitates troubleshooting and performance analysis.
[0033] In this embodiment, the microservice structure system can implement multiple different types of business requests. In order to classify and store the log data corresponding to different types of business requests generated in actual business, the corresponding subject identification information can be configured for different types of business. The subject identification information is a keyword or phrase used to describe, classify or mark various types of business requests.
[0034] The request method refers to the type of operation defined in the communication protocol. For example, the request method may include: GET, which is used to request the server to send resources and is usually used to obtain data. For example, obtain web page content from the server. POST, which is used to submit data to the server and is usually used to create resources. For example, submit form data to the server. PUT, which is used to update or replace resources. For example, update data on the server. DELETE, which is used to delete resources. For example, delete a file or record from the server. CONNECT, which establishes a tunnel connection, is usually used for SSL encrypted communication. OPTIONS: Returns the HTTP request methods supported by the server. TRACE, which is used to trace the transmission path of the request-response. In particular, annotation information can be added to the request method. Annotation is a mechanism for adding supplementary information to the source code, which can be used to describe the characteristics and behaviors of program elements such as classes, methods, and fields. Annotation information does not affect the actual logic of the code, but can be parsed and used by the compiler and runtime tools to serve the functions of description and configuration. In this embodiment, the purpose of adding annotation information to the request method is that some business requests are more important to the actual business, and custom annotation information can be added to the request methods of these business requests. When the system recognizes these custom annotation information, it can classify and store the link logs corresponding to these business requests through a pre-set log data storage method; while some business requests are not very important to the actual business, and annotation information may not be added to the request methods of these business requests, or annotation information corresponding to the default processing method may be added, and the link log data corresponding to such business requests can be processed according to the default log data processing method.
[0035] When the client generates a target service request, the target service request carries at least one service unit involved in the target service request, so that the identification information corresponding to these service units is easy to determine. The request information corresponding to each service unit may include the request IP, request port, specific request method, request parameters, and start time, etc.
[0036] In this embodiment, a specific implementation method for determining the log object information corresponding to the target business request may include: when receiving the target business request, intercepting the target business request, obtaining basic request information carried in the target business request, wherein the basic request information includes a business request category, a request method including annotation information, identification information of at least one service unit involved in the request, and request information corresponding to each service unit; based on the business request category in the basic request information, determining the subject identification information corresponding to the business request category; generating a unique link identifier corresponding to the target business request; and assembling the basic request information, the unique link identifier, and the subject identification information into one log object information.
[0037] Specifically, when a target business request is received from a client, the target business request can be intercepted first, and then the target business request can be parsed to obtain the business request category carried in the target business request, the request method containing annotation information, the identification information of each service unit involved in the target business request, and the request information corresponding to each service unit. Further, according to the business request category and the mapping relationship between the pre-defined various types of business and subject identifiers, the subject identification information corresponding to the target business request is determined. Furthermore, a corresponding unique identification code can be generated for the target business request, and this unique identification code is used as a unique link identifier. Based on this, the above-determined unique link identifier, subject identification information, request method containing annotation information, identification information of each service unit involved in the target business request, and request information corresponding to each service unit are assembled into an overall log object information.
[0038] Based on the above embodiment, after obtaining the log object information corresponding to the target business request, the current thread corresponding to the target business request can be called, and the log object information can be placed in the current thread data structure corresponding to the target business request, so as to execute the link log storage method corresponding to the target business request based on the current thread.
[0039] In this embodiment, a thread can be called to process the service response step and link log generation and storage step corresponding to the target service request, and this thread is the current thread. By putting the log object information into the data structure of the current thread, the current thread can execute the service corresponding to the target service request and the link log storage method. After that, whenever a target service request is received from a client, a corresponding thread can be called to execute the corresponding data operation process.
[0040] S120 , making a service request to each service unit based on the log object information, and determining target link log data corresponding to the target service request when each service unit completes the corresponding service.
[0041] The target link log data is the link log data corresponding to the target service request. By recording the link log, we can record and track the entire process of a request from start to finish, including the calling and transmission of the request between various service units, which can help us monitor the status of the current request in detail and quickly determine the specific location where the error occurred.
[0042] In this embodiment, the service corresponding to each service unit can be executed according to the request method in the log object information and the request information corresponding to each service unit. At the same time, when each service unit executes the corresponding business service, each service unit can generate corresponding business log data, and the unique link identifier in the log object information can be added to the business log generated by each service unit to obtain the link log data. After each service unit has executed the corresponding business service, the total response time corresponding to the target business service is obtained, that is, the total time, request result, exception stack data and link log data for each service unit to execute the corresponding business service. Further, the obtained data content is supplemented to the log object information to obtain the target link log data corresponding to the target business request.
[0043] S130. When the annotation information included in the request method meets the preset annotation condition, the target link log data is stored in the target distributed system by means of a message queue based on the subject identification information.
[0044] The preset annotation condition is a pre-set rule for determining whether to store the target link log data in the target distributed system. In this embodiment, the preset annotation condition is that the annotation information includes the target field content, and the target field content is used to indicate that the target link log data corresponding to the target business request needs to be specially stored.
[0045] Among them, the distributed system is composed of a master node and several slave nodes. It is an important part of the cluster and has been widely used in the field of large-scale computing due to its reliability, efficiency and scalability. The target distributed system is the distributed system to which the target link log data is uploaded. It should be noted that the target distributed system can include multiple logical storage units, each of which is used to store a document collection with the same structure.
[0046] In this embodiment, the target link log data can be stored in the target distributed system in an orderly manner through the message queue. For example, the method of the KAFKA distributed message queue can be used to implement the data warehousing task. Using the publish-subscribe mode of KAFKA, the KAFKA method is mainly composed of three types of roles: message producer (Producer), proxy server (Broker) and message consumer (Consumer). Among them, if multiple proxy servers are set, it can be called a proxy server cluster. The message producer generates a message of a specified topic (Topic) and passes it to the proxy server cluster. The proxy server cluster maintains various Topic message queues on the disk storage. The message consumer who subscribes to a Topic pulls the newly generated message from the proxy server cluster and processes it. The advantage of this pull method is that consumers can independently control the message rate to avoid the message backlog problem caused by the slow processing speed of consumers.
[0047] In this embodiment, the annotation information in the request method can be obtained. If the target field content is included in the annotation information, the target link log data can be classified and stored in the target distributed system through the message middleware in the form of a message queue, with the subject identification information as the name information of the logical storage unit. In addition, if the target field content is not included in the annotation information, it can be processed according to the default log data processing method, for example, the default log data processing method is to directly print the corresponding log data, or store the corresponding log data in a preset storage space.
[0048] It is understandable that the usual log processing method cannot clearly distinguish between service operation logs, business logs and link logs. Link logs of multiple services are stored together with business logs and operation logs in one database, which is not conducive to the classified storage and data migration of log data. In this step, link log data is collected asynchronously using the publish-subscribe mode of the message queue through the subject identification information, which can separate the link log from other types of log data and optimize the log storage mode.
[0049] The technical solution of the embodiment of the present invention is to determine the log object information corresponding to the target business request when receiving the target business request, wherein the log object information includes a unique link identifier, subject identifier information, a request method containing annotation information, identifier information of at least one service unit involved in the request, and request information corresponding to each service unit, and then, based on the log object information, a service request is made to each service unit, and when each service unit completes the corresponding service, the target link log data corresponding to the target business request is determined, so that, when the annotation information contained in the request method meets the preset annotation condition, the target link log data is stored in the target distributed system by means of a message queue based on the subject identifier information. The technical solution of this embodiment can generate link log data through a unique link identifier, and can associate all service log data involved in a request according to the sequence of the entire steps, and can efficiently locate the fault link according to the link log, thereby improving the efficiency of service fault troubleshooting.
[0050] Embodiment 2
[0051] Figure 2 This is a schematic diagram of a link log data processing method provided by an embodiment of the present invention. Based on the above embodiment, S120 and S130 are further refined. The specific implementation method can refer to the technical solution of this embodiment. Among them, the technical terms that are the same or corresponding to the above embodiment are not repeated here.
[0052] like Figure 2 As shown, the method specifically comprises the following steps:
[0053] S210. When a target business request is received, determine the log object information corresponding to the target business request, wherein the log object information includes a unique link identifier, subject identifier information, a request method including annotation information, identifier information of at least one service unit involved in the request, and request information corresponding to each service unit.
[0054] S220: Execute the business service corresponding to each service unit based on the request method, identification information of at least one service unit involved in the request, and request information corresponding to each service unit.
[0055] In this embodiment, based on the identification information of at least one service unit involved in the request, it can be determined which service units will be relied upon to perform the business services, so that the business services corresponding to each service unit can be executed in sequence according to the request method and the request information corresponding to each service unit, such as the request IP, request port, specific request method, request parameters and start time.
[0056] S230: When each service unit performs a corresponding business service, a unique link identifier is added to a business log generated by each service unit to obtain link log data.
[0057] In this embodiment, each service unit can generate corresponding business data when executing the corresponding business service, and can add the unique link identifier to the business log generated by each service unit. By transmitting the unique link identifier, all business logs corresponding to the target business request can be associated, thereby obtaining the link log data corresponding to the entire service link. By using the unique link identifier, the business log data generated by multiple service units can be associated with each other, thereby improving the efficiency of service fault troubleshooting.
[0058] S240: When each service unit completes the service request, request feedback data is obtained.
[0059] Among them, the request feedback data includes at least one of link log data, total request response time, request result and exception stack information. The total request response time refers to the total response time corresponding to the target business request. The request result refers to the final result corresponding to the entire service link and / or the request sub-result corresponding to each service unit. Exception stack information refers to the error information automatically recorded by the system when an exception occurs during program execution, including the specific location of the exception, the method call sequence that caused the exception, the cause of the exception, and other information. These exception stack information are very helpful for developers to debug and locate problems.
[0060] In this embodiment, when each service unit completes the service, link log data, request response total duration, request result and exception stack information can be generated. Based on this, these data contents can be obtained as request feedback data.
[0061] S250: Supplement the request feedback data into the log object information to obtain the target link log data corresponding to the target service request.
[0062] In this embodiment, on the basis of obtaining the request feedback data, these data contents are supplemented to the log object information determined in S210, and the target link log data corresponding to the target service request can be obtained. The purpose of this setting is to obtain the target link log data by generating log information after the request and supplementing the log object information, thereby enriching the monitoring content of the entire link log.
[0063] S260. When the annotation information included in the request method meets the preset annotation condition, the target link log data is published as the current event to the target message queue corresponding to the subject identification information.
[0064] In this embodiment, the target link log data can be stored in the target distributed system in an orderly manner through the message middleware. The message middleware can include message queues corresponding to multiple topics, and the target message queue is the message queue corresponding to the topic identification information of the target service request.
[0065] In this embodiment, if the annotation information contained in the request method meets the preset annotation conditions, after obtaining the target link log data corresponding to the target business request, the target link log data can be published to the message middleware as the current event. Thus, according to the subject identification information, the target link log data can be added to the target message queue corresponding to the subject identification information in the message middleware.
[0066] In this embodiment, dynamic confirmation of whether to store link log data is achieved through annotation information, which ensures the flexible configuration of link log collection function and reduces development costs. Link logs are sent and stored asynchronously through message queues, which ensures application performance, optimizes log storage mode, realizes separate storage of link logs, and improves storage utilization of data storage systems.
[0067] S270: When the target link log data in the target message queue meets a preset condition, the target link log data is classified and stored in the target distributed system using the subject identification information as the name information of the logical storage unit.
[0068] The preset condition is a pre-set condition. When the data in the target message queue reaches the preset condition, the data in the target message queue can be classified and stored in the target distributed system. Optionally, the preset condition includes at least one of the following: the amount of data in the target message queue reaches a preset data amount threshold; the waiting time in the target message queue reaches a preset waiting time threshold.
[0069] In this embodiment, when determining whether to store the target link log data in the target message queue in the target distributed system, the following judgment criteria may be included. It may be that the amount of data in the target message queue reaches a preset data amount threshold, for example, the data in the target message queue reaches 2000, at which time the data in the target message queue can be stored in the target distributed system based on the warehousing process; it may also be that the waiting time of the data in the target message queue reaches a preset waiting time threshold. For example, the waiting time of the data in the target message queue reaches 2 minutes, at which time the data in the target message queue can be stored in the target distributed system based on the warehousing process; it may also be that the data in the target message queue reaches 2000, and the waiting time of the data in the target message queue reaches 2 minutes, and the data in the target message queue can be stored in the target distributed system based on the warehousing process. The warehousing process will splice and convert batch warehousing statements according to the configuration file of the target distributed system table field, and generate warehousing statement conversion. For example, it supports batch writing of commonly used target distributed systems on the market such as ORACLE, MYSQL, SQLITE, POSTGRE, etc., to realize batch warehousing of data and further improve data warehousing efficiency.
[0070] Based on the above embodiments, the target distributed system may include a large number of logical storage units, and the target link log data may be stored in a logical storage unit in the target distributed system, and the subject identification information corresponding to the target business request may be used as the name information of the logical storage unit, thereby achieving the purpose of classifying and storing the link log data.
[0071] Based on the above embodiment, before storing the target link log data in the target distributed system through a message queue based on the subject identification information, it can also include: formatting the target link log data based on a preset processing method, so as to classify and store the formatted target link log data in the target distributed system.
[0072] The preset processing method is a pre-set data processing method used to format the target link log data. For example, the preset processing method may be to remove useless data from the target link log data according to a preset removal condition; or it may be to unify the data format of the target link log data into a preset data format.
[0073] In this embodiment, before the target link log data is classified and stored, the target link log data may be formatted according to a preset processing method to improve the standardization and effectiveness of the target link log data.
[0074] For example, the link log data processing method provided by this embodiment is described below with a specific implementation method. Figure 3 , mainly including the following steps:
[0075] Step 1: When receiving a target service request from the server, the interceptor intercepts the target request and generates log object information corresponding to the target service request through the pre-method in the interceptor; wherein the log object information includes a unique link identifier, subject identifier information, a request method containing annotation information, identifier information of at least one service unit involved in the request, and request information corresponding to each service unit. Step 2: Put the generated log object information into the data structure corresponding to the current thread and start the link request. Step 3: When each service unit completes the corresponding service, the interceptor takes out the information of the log object stored in the data structure corresponding to the current thread, supplements the link log data, the total duration of the request response, the request result, and the exception stack information, and obtains the target link log data. Step 4: Obtain the annotation information on the request method. If the annotation information meets the preset annotation conditions, the target link log data will be published as an event to the message middleware through the subject identifier information. Step 5: Listen to the message event through the listener, and when the message event is listened, send the message event to the message middleware. Step 6: The message collector uses the message middleware as the data source, consumes the data in the message middleware, and formats the target link log data. Step 7: For the formatted target link log data, the subject identification information is used as the name information of the logical storage unit, and the target link log data is classified and stored in the target distributed system.
[0076] The technical solution of the embodiment of the present invention, when determining the target link log data, executes the business service corresponding to each service unit based on the request method, the identification information of at least one service unit involved in the request, and the request information corresponding to each service unit. Then, when each service unit executes the corresponding business service, the unique link identifier is added to the business log generated by each service unit to obtain the link log data. Thus, when each service unit completes the service request, the request feedback data is obtained, and the request feedback data includes at least one of the link log data, the total duration of the request response, the request result, and the exception stack information. Subsequently, the request feedback data is supplemented to the log object information to obtain the target link log data corresponding to the target business request. The technical solution provided by this embodiment obtains the target link log data by generating log information after the request and supplementing the log object information, thereby enriching the monitoring content of the entire link log; at the same time, by using the unique link identifier, the mutual correlation of the business log data generated by multiple service units is achieved, thereby improving the efficiency of service troubleshooting. Furthermore, when the annotation information included in the request method meets the preset annotation conditions, the target link log data is published as the current event to the target message queue corresponding to the subject identification information, and then, when the target link log data in the target message queue meets the preset conditions, the subject identification information is used as the name information of the logical storage unit, and the target link log data is classified and stored in the target distributed system. The technical solution provided by this embodiment dynamically confirms whether to store link log data through annotation information, ensures the flexible configuration of the link log collection function, and also reduces the development cost; at the same time, the link log is sent and stored asynchronously through the message queue, which ensures the performance of the application, optimizes the storage mode of the log, realizes the separate storage of the link log, and improves the storage utilization of the data storage system.
[0077] Embodiment 3
[0078] Figure 4 This is a schematic diagram of the structure of a link log data processing device provided by an embodiment of the present invention. The device includes: a log object determination module 310, a link log determination module 320 and a link log storage module 330.
[0079] The log object determination module 310 is used to determine the log object information corresponding to the target business request when receiving the target business request; wherein the log object information includes a unique link identifier, subject identifier information, a request method including annotation information, identifier information of at least one service unit involved in the request, and request information corresponding to each service unit;
[0080] The link log determination module 320 is used to make a service request to each service unit based on the log object information, and determine the target link log data corresponding to the target service request when each service unit completes the corresponding service;
[0081] The link log storage module 330 is used to store the target link log data in the target distributed system through a message queue based on the subject identification information when the annotation information included in the request method meets the preset annotation condition.
[0082] Based on the above device, optionally, the log object determination module 310 includes:
[0083] A request information acquisition unit, configured to intercept the target business request when receiving the target business request, and acquire basic request information carried in the target business request; wherein the basic request information includes a business request category, a request method including annotation information, identification information of at least one service unit involved in the request, and request information corresponding to each service unit;
[0084] A subject identification determination unit, configured to determine subject identification information corresponding to the service request category based on the service request category in the basic request information;
[0085] A link identifier generating unit, used to generate a unique link identifier corresponding to a target service request;
[0086] The log object assembly unit is used to assemble the basic request information, the unique link identifier and the subject identifier information into a log object information.
[0087] Based on the above-mentioned device, optionally, a link log data processing device includes: a current thread processing module, which is used to call the current thread corresponding to the target business request, and put the log object information into the data structure corresponding to the current thread, so as to execute the link log storage method corresponding to the target business request based on the current thread.
[0088] Based on the above device, optionally, the link log determination module 320 includes:
[0089] A service execution unit, configured to execute a business service corresponding to each service unit based on a request method, identification information of at least one service unit involved in the request, and request information corresponding to each service unit;
[0090] The link log data determination unit is used to add a unique link identifier to the business log generated by each service unit when each service unit performs a corresponding business service, so as to obtain link log data.
[0091] Based on the above device, optionally, the link log determination module 320 further includes:
[0092] A feedback data acquisition unit, used to acquire request feedback data when each service unit completes a service request; wherein the request feedback data includes at least one of link log data, total request response time, request result, and exception stack information;
[0093] The target link data determination unit is used to supplement the request feedback data into the log object information to obtain the target link log data corresponding to the target service request.
[0094] Based on the above device, optionally, the link log storage module 330 includes:
[0095] A log data queuing unit, used to publish the target link log data as a current event to a target message queue corresponding to the subject identification information when the annotation information contained in the request method meets a preset annotation condition;
[0096] A link log storage unit is used to classify and store the target link log data in a target distributed system using subject identification information as the name information of the logical storage unit when the target link log data in the target message queue meets preset conditions; wherein the preset conditions include at least one of the following: the amount of data in the target message queue reaches a preset data amount threshold; the waiting time in the target message queue reaches a preset waiting time threshold.
[0097] Based on the above device, optionally, the link log data processing device also includes: a formatting processing module, which is used to format the target link log data based on a preset processing method, so as to classify and store the formatted target link log data in the target distributed system.
[0098] The technical solution of the embodiment of the present invention is to determine the log object information corresponding to the target business request when receiving the target business request, wherein the log object information includes a unique link identifier, subject identifier information, a request method containing annotation information, identifier information of at least one service unit involved in the request, and request information corresponding to each service unit, and then, based on the log object information, a service request is made to each service unit, and when each service unit completes the corresponding service, the target link log data corresponding to the target business request is determined, so that, when the annotation information contained in the request method meets the preset annotation condition, the target link log data is stored in the target distributed system by means of a message queue based on the subject identifier information. The technical solution of this embodiment can generate link log data through a unique link identifier, and can associate all service log data involved in a request according to the sequence of the entire steps, and can efficiently locate the fault link according to the link log, thereby improving the efficiency of service fault troubleshooting.
[0099] The link log data processing device provided in the embodiment of the present invention can execute the link log data processing method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.
[0100] It is worth noting that the various units and modules included in the above system are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the protection scope of the embodiments of the present invention.
[0101] Embodiment 4
[0102] Figure 5 The present invention is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. Figure 5 A block diagram of an exemplary electronic device 40 suitable for implementing exemplary embodiments of the present invention is shown. Figure 5 The electronic device 40 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.
[0103] like Figure 5 As shown, the electronic device 40 is in the form of a general computing device. The components of the electronic device 40 may include but are not limited to: one or more processors or processing units 401, a system memory 402, and a bus 403 connecting different system components (including the system memory 402 and the processing unit 401).
[0104] Bus 403 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor or a local bus using any of a variety of bus structures. For example, these architectures include but are not limited to Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MAC) bus, Enhanced ISA bus, Video Electronics Standards Association (VESA) local bus and Peripheral Component Interconnect (PCI) bus.
[0105] The electronic device 40 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the electronic device 40, including volatile and non-volatile media, removable and non-removable media.
[0106] System memory 402 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 404 and / or cache memory 405. Electronic device 40 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 406 may be used to read and write non-removable, non-volatile magnetic media ( Figure 5 not shown, usually called a "hard drive"). Although Figure 5 Not shown in the figure, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk"), and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, a DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to bus 403 via one or more data medium interfaces. Memory 402 may include at least one program product having a set (e.g., at least one) of program modules that are configured to perform the functions of various embodiments of the present invention.
[0107] A program / utility 408 having a set (at least one) of program modules 407 may be stored, for example, in memory 402, such program modules 407 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment. Program modules 407 generally perform the functions and / or methods of the embodiments described herein.
[0108] The electronic device 40 may also communicate with one or more external devices 409 (e.g., keyboards, pointing devices, displays 810, etc.), may communicate with one or more devices that enable a user to interact with the electronic device 40, and / or may communicate with any device that enables the electronic device 40 to communicate with one or more other computing devices (e.g., a network card, a modem, etc.). Such communication may be performed via an input / output (I / O) interface 411. Furthermore, the electronic device 40 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 412. As shown, the network adapter 412 communicates with other modules of the electronic device 40 via the bus 403. It should be understood that although Figure 5 Not shown, other hardware and / or software modules may be used in conjunction with the electronic device 40, including but not limited to microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.
[0109] The processing unit 401 executes various functional applications and page processing by running the programs stored in the system memory 402, for example, implementing the link log data processing method provided in the embodiment of the present invention.
[0110] Embodiment 5
[0111] An embodiment of the present invention further provides a storage medium containing computer executable instructions, which are used to execute a link log data processing method when executed by a computer processor. The method includes:
[0112] When a target service request is received, log object information corresponding to the target service request is determined; wherein the log object information includes a unique link identifier, subject identifier information, a request method including annotation information, identifier information of at least one service unit involved in the request, and request information corresponding to each service unit;
[0113] Performing service requests on each service unit based on the log object information, and determining target link log data corresponding to the target service request when each service unit completes the corresponding service;
[0114] When the annotation information included in the request method meets the preset annotation conditions, the target link log data is stored in the target distributed system through a message queue based on the subject identification information.
[0115] The computer storage medium of the embodiment of the present invention can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, a computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it.
[0116] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.
[0117] Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
[0118] Computer program code for performing the operation of embodiments of the present invention may be written in one or more programming languages or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).
[0119] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.
Claims
1. A link log data processing method, characterized in that: include: When a target service request is received, determining log object information corresponding to the target service request; wherein the log object information includes a unique link identifier, subject identifier information, a request method including annotation information, identifier information of at least one service unit involved in the request, and request information corresponding to each of the service units; Performing a service request on each of the service units based on the log object information, and determining target link log data corresponding to the target service request when each of the service units completes a corresponding service; In the case where the annotation information included in the request method meets a preset annotation condition, the target link log data is stored in a target distributed system by way of a message queue based on the subject identification information.
2. The method according to claim 1, characterized in that When receiving a target service request, determining log object information corresponding to the target service request includes: When receiving a target service request, intercept the target service request and obtain basic request information carried in the target service request; wherein the basic request information includes a service request category, a request method including annotation information, identification information of at least one service unit involved in the request, and request information corresponding to each of the service units; Based on the service request category in the basic request information, determining subject identification information corresponding to the service request category; Generate a unique link identifier corresponding to the target service request; The basic request information, the unique link identifier and the subject identifier information are assembled into a log object information.
3. The method according to claim 1, characterized in that Before making a service request to each of the service units based on the log object information, the method further includes: The current thread corresponding to the target service request is called, and the log object information is placed in a data structure corresponding to the current thread, so as to execute a link log storage method corresponding to the target service request based on the current thread.
4. The method according to claim 1, characterized in that The performing a service request on each of the service units based on the log object information includes: Based on the request method, identification information of at least one service unit involved in the request, and request information corresponding to each of the service units, executing the business service corresponding to each of the service units; When each of the service units executes the corresponding business service, the unique link identifier is added to the business log generated by each of the service units to obtain link log data.
5. The method according to claim 4, characterized in that When each of the service units completes a corresponding service, determining the target link log data corresponding to the target service request includes: When each of the service units completes the service request, request feedback data is obtained; wherein the request feedback data includes at least one of the link log data, the total request response time, the request result, and the exception stack information; The request feedback data is added to the log object information to obtain the target link log data corresponding to the target service request.
6. The method according to claim 1, characterized in that When the annotation information included in the request method satisfies a preset annotation condition, based on the subject identification information, the target link log data is stored in a target distributed system by means of a message queue, including: In the case where the annotation information contained in the request method meets the preset annotation condition, publishing the target link log data as the current event to the target message queue corresponding to the subject identification information; When the target link log data in the target message queue meets the preset conditions, the target link log data is classified and stored in the target distributed system using the subject identification information as the name information of the logical storage unit; wherein the preset conditions include at least one of the following: the amount of data in the target message queue reaches a preset data amount threshold; the waiting time in the target message queue reaches a preset waiting time threshold.
7. The method according to claim 6, characterized in that Before storing the target link log data in the target distributed system by means of a message queue based on the subject identification information, the method further includes: The target link log data is formatted based on a preset processing method, so that the formatted target link log data is classified and stored in a target distributed system.
8. A link log data processing device, characterized in that: The device includes: A log object determination module, configured to determine log object information corresponding to a target business request when receiving the target business request; wherein the log object information includes a unique link identifier, subject identifier information, a request method including annotation information, identifier information of at least one service unit involved in the request, and request information corresponding to each of the service units; A link log determination module, configured to make a service request to each of the service units based on the log object information, and determine target link log data corresponding to the target service request when each of the service units completes the corresponding service; The link log storage module is used to store the target link log data in the target distributed system through a message queue based on the subject identification information when the annotation information contained in the request method meets the preset annotation condition.
9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the link log data processing method according to any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the link log data processing method according to any one of claims 1 to 7 when executed.