Data processing method and device, equipment, storage medium and program product

Abstract

The embodiment of the invention provides a data processing method and device, equipment, a storage medium and a program product, and is applied to the technical field of databases. The method comprises the steps that according to a task request input by a user, a target identifier of the task request is generated, and the target identifier is related to identifier generation time, a working node and a task sequence. And when the task request enters a service, configuring the target identifier to a target thread for executing the task request. And when a database message corresponding to the task request is transmitted through the target thread, adding the target identifier to metadata of the database message to obtain an extended database message. And executing a target operation in the target database based on the extended database message. According to the method provided by the invention, the tracking continuity of link tracking and performance monitoring of the distributed system and the accuracy and effectiveness of fault positioning and performance optimization can be improved.

Description

Technical Field

[0001] This application relates to the field of database technology, and in particular to a data processing method, apparatus, device, storage medium, and program product. Background Technology

[0002] In distributed systems, tracing and performance monitoring are crucial for ensuring stable system operation, quickly locating faults, and optimizing performance. By recording and analyzing the complete path of requests within a multi-node service environment in a distributed system, bottlenecks and abnormal nodes in the request processing can be accurately identified, and the call relationships and time consumption of each service can be clearly understood, providing a strong basis for system performance tuning and troubleshooting. However, current technologies cannot effectively transmit link identifiers during the database access phase, resulting in a disconnect between database execution information and upper-layer requests. This makes it difficult to achieve unified end-to-end tracing and performance analysis, leading to problems such as poor tracing continuity and low accuracy and effectiveness in fault location and performance optimization in distributed system tracing and performance monitoring.

[0003] Therefore, improving the continuity of link tracing and performance monitoring, as well as the accuracy and effectiveness of fault location and performance optimization, are urgent problems that need to be solved. Summary of the Invention

[0004] The data processing methods, apparatus, devices, storage media, and program products provided in this application are used to improve the continuity of link tracing and performance monitoring, as well as the accuracy and effectiveness of fault location and performance optimization.

[0005] In a first aspect, embodiments of this application provide a data processing method, including:

[0006] Based on the task request input by the user, a target identifier for the task request is generated, and the target identifier is related to the identifier generation time, work node, and task sequence.

[0007] When the task request enters the service, the target identifier is configured in the target thread that executes the task request;

[0008] When transmitting the database message corresponding to the task request through the target thread, the target identifier is added to the metadata of the database message to obtain an extended database message;

[0009] The target operation is performed in the target database based on the extended database message.

[0010] Optionally, configuring the target identifier into the target thread that executes the task request includes:

[0011] The target identifier is obtained by intercepting the task request;

[0012] Configure the target identifier into the context of the target thread.

[0013] Optionally, adding the target identifier to the metadata of the database message to obtain an extended database message includes:

[0014] The target identifier is added to an extended field in the metadata of the database message to generate the extended database message, wherein the extended field includes the value of the target identifier.

[0015] Optionally, the extended field may also include an extended field identifier field and a target identifier length field.

[0016] Optionally, generating the target identifier of the task request based on the user-input task request includes:

[0017] Obtain the worker node identifier corresponding to the task request and the task sequence identifier of the task request;

[0018] The target identifier is generated based on the preset symbol bit, the working node identifier, the task sequence identifier, and the current time.

[0019] Optionally, generating the target identifier based on the preset symbol bit, the working node identifier, the task sequence identifier, and the current time includes:

[0020] When the current time is less than the generation time of the previous identifier, and when the current time is greater than or equal to the generation time of the previous identifier, the target identifier is generated according to the preset symbol bit, the working node identifier, the task sequence identifier, and the current time.

[0021] Optionally, the method further includes:

[0022] Add the target identifier to the response message of the target database;

[0023] And / or,

[0024] When the target database executes the SQL statement in the database message, the SQL statement is associated with the target identifier and stored in the database log.

[0025] Secondly, embodiments of this application provide a data processing apparatus, including:

[0026] The first processing module is used to generate a target identifier for the task request based on the task request input by the user. The target identifier is related to the identifier generation time, working node, and task sequence.

[0027] The first control module is used to configure the target identifier into the target thread that executes the task request when the task request enters the service;

[0028] The second control module is used to add the target identifier to the metadata of the database message when transmitting the database message corresponding to the task request through the target thread, so as to obtain an extended database message.

[0029] The third control module is used to perform target operations in the target database based on the extended database message.

[0030] Thirdly, embodiments of this application provide an electronic device, including: a memory and a processor;

[0031] The memory stores computer-executed instructions;

[0032] The processor executes computer execution instructions stored in the memory, causing the processor to perform the first aspect and / or various possible implementations of the first aspect as described above.

[0033] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the first aspect and / or various possible implementations of the first aspect.

[0034] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the first aspect and / or various possible implementations of the first aspect.

[0035] The data processing method, apparatus, device, storage medium, and program product provided in this application generate a target identifier related to the identifier generation time, working node, and task sequence of a task request based on user-input task request. When the task request enters the service, the target identifier is configured into the target thread executing the task request. When transmitting the database message corresponding to the task request through the target thread, the target identifier is added to the metadata of the database message to obtain an extended database message. The target operation is then executed in the target database based on the extended database message, thereby achieving effective transmission of the link identifier during the database access phase. This ensures that the database execution information is closely associated with the upper-layer request, achieving end-to-end complete link tracing and accurate performance analysis. Compared with the prior art, this overcomes the problem of difficult link identifier transmission during the database access phase, ensuring uninterrupted tracing, providing more accurate and comprehensive information for fault location, and providing stronger data support for performance optimization. This improves the continuity of link tracing and performance monitoring, as well as the accuracy and effectiveness of fault location and performance optimization. Attached Figure Description

[0036] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0037] Figure 1 This is a structural diagram illustrating the application scenario of this application;

[0038] Figure 2 A flowchart illustrating a data processing method provided in an embodiment of this application;

[0039] Figure 3 A flowchart illustrating another data processing method provided in an embodiment of this application;

[0040] Figure 4 A flowchart illustrating another data processing method provided in an embodiment of this application;

[0041] Figure 5 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application;

[0042] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0043] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0044] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0045] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented, for example, in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0046] Currently, existing distributed tracing systems can achieve a certain degree of request tracing and cross-service performance monitoring at the application layer by attaching a trace identifier (TraceId) to the request, such as passing TraceId and SpanId between service calls, or collecting call information at the application layer and reporting it to the tracing server.

[0047] However, these solutions generally rely on injecting a link identifier into HyperText Transfer Protocol (HTTP) or RPC calls. During the database access phase, existing technologies cannot effectively transmit the link identifier, causing database execution information to become disconnected from upper-layer requests, making it difficult to achieve unified end-to-end tracing and performance analysis.

[0048] Existing solutions based on Structured Query Language (SQL) annotations have significant technical limitations: on the one hand, when the database server enables statement caching, the preprocessed handle generated during the initial compilation no longer carries the original SQL text containing the TracerId during subsequent executions, causing tracing interruptions; on the other hand, modifying the SQL statement structure may not only affect the behavior of the database optimizer but may also trigger security audit issues.

[0049] In view of this, this application provides a data processing method that generates a target identifier for the task request based on the user-input task request, which is related to the identifier generation time, working node, and task sequence. When the task request enters the service, the target identifier is configured into the target thread executing the task request. When transmitting the database message corresponding to the task request through the target thread, the target identifier is added to the metadata of the database message to obtain an extended database message. The target operation is then executed in the target database based on the extended database message. This achieves effective transmission of the link identifier during the database access phase, closely linking database execution information with the upper-layer request, and achieving end-to-end complete link tracing and accurate performance analysis. Compared with existing technologies, this method overcomes the difficulty of link identifier transmission during the database access phase, ensuring uninterrupted tracing, providing more accurate and comprehensive information for fault location, and providing stronger data support for performance optimization. This improves the continuity of link tracing and performance monitoring, as well as the accuracy and effectiveness of fault location and performance optimization.

[0050] Figure 1 This is a structural diagram illustrating the application scenario of this application, as shown below. Figure 1 As shown, this application is applied in the field of database technology, and the scenario includes databases, distributed systems, and clients.

[0051] The database can be a device that can store data, such as an SQL database, an Oracle database, or a Kingbase database. This application does not limit the type of database.

[0052] The client is used to provide an interface for users, receive data operation instructions (such as data insertion, query, update, deletion, etc.) input by users, and send these instructions to the database through a distributed system (i.e., the server); at the same time, it receives information such as data query results returned by the database and presents it to the user, realizing the interaction between the user and the database system.

[0053] A distributed system comprises multiple worker nodes to provide service entry points to clients. For example, a distributed system may include multiple data centers, each containing multiple machines acting as worker nodes. When a user accesses the service through a client, they can connect to the database through any worker node in the distributed system.

[0054] The database host receives task requests from clients via the distributed system. These requests may include data manipulation instructions. The host then processes the data in the database according to these instructions, such as performing data insertion operations to store new data in a specified location within the database, or performing query operations to retrieve data that meets certain criteria from the database. The processed results are then returned to the client.

[0055] In this scenario, users can interact with data through the client-distributed system-database link, such as storing new data in the database host, performing data insertion operations to store new data in a specified location in the database, and performing query operations to retrieve data that meets the conditions from the database.

[0056] Below, with Figure 1 Taking the system structure shown as an example, the technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems will be described in detail through specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will be described below with reference to the accompanying drawings.

[0057] Figure 2 This is a flowchart illustrating a data processing method provided in an embodiment of this application. Figure 2 As shown, the method includes:

[0058] S201. Generate the target identifier of the task request based on the task request input by the user.

[0059] The user-input task request is an access request to the business database, carrying information such as request type, data operation scope, and the identity of the request initiator. The target identifier is a globally unique link identifier for the task request within the distributed system, such as a TracerId. The target identifier can be a fixed-length field, such as a 64-bit TracerId.

[0060] The target identifier can be generated by processing relevant information from the task request, such as through the snowflake algorithm, an improved snowflake algorithm, an algorithm based on a combination of timestamps and random numbers, or a distributed ID generation algorithm based on database auto-incrementing ID extension. Alternatively, it can be generated using the global clock service of the distributed system itself, combined with the unique identifier of the node and the specific characteristics of the task, through specific encoding rules. Specifically, the target identifier can be generated on the entry service node of the distributed system to uniquely identify the target task in the execution chain.

[0061] For example, when an entry service node receives a task request, it generates a target identifier using an improved snowflake algorithm. This target identifier can be related to the identifier generation time, the worker node, and the task sequence. For instance, the entry service node can concatenate a preset symbol bit, the identifier generation time, the worker node's identifier, and the task sequence number to generate the target identifier. Assuming the preset sign bit is 0, the identifier generation time is 000000000000000000000000000000000000111010101111011 (representing the time difference from the initial time), the worker node identifier is 0010100011 (which can represent the machine identifier corresponding to the data center and the worker node), and the task sequence number is 000000001100 (which can represent the concurrent sequence number of concurrent tasks generated at the same time), then the target identifier is 0 000000000000000000000000000000000000111010101111011 0010100011 000000001100.

[0062] S202. When a task request enters the service, the target identifier is configured in the target thread that executes the task request.

[0063] In this embodiment, "task request entering service" refers to the process of a business database access request being integrated into the database service's processing flow. The process of entering the service includes steps such as request reception, legality verification, and request classification. Legality verification may include verifying the identity information of the request initiator and verifying the permission of the request operation type. Request classification can allocate the request to the corresponding processing queue according to the scope of data operation.

[0064] Once a task request enters the service, a corresponding target thread can be assigned to it. The target thread has an independent execution space and resource allocation quota. The thread's execution space includes a pre-defined area for storing various related information, including a dedicated area for storing the target identifier. For example, this area can be implemented based on the ThreadLocal mechanism, ensuring the uniqueness and accessibility of the target identifier within the current thread.

[0065] Then, the target identifier is configured to the target thread. For example, the address information of the area in the target thread used to store the identifier can be determined first. This address information is associated with the execution control information of the thread and can be obtained through the thread's basic attributes. Then, the multi-dimensional identifier corresponding to the target identifier is written to the storage area corresponding to the address. During the writing process, the dimension codes can be stored in the order of the dimensions to ensure that the written data can be completely read by subsequent operations.

[0066] One possible implementation is to intercept the target identifier from the task request and then configure the target identifier in the context of the target thread. This method can accurately obtain the target identifier corresponding to the current task request through preset interception rules, avoiding identifier confusion. Specifically, this can be achieved through the aforementioned bytecode-enhanced interception mechanism, where the interceptor, after obtaining the target identifier from the task request, directly writes it to the thread's ThreadLocal storage area.

[0067] Another possible implementation is to obtain the target identifier corresponding to the current task request from the source of identifier generation. In asynchronous execution or cross-thread scheduling scenarios, the target identifier can be passed between different execution threads through context copying or parameter passing to ensure the consistency of the target identifier throughout the entire task request process.

[0068] S203. When transmitting the database message corresponding to the task request through the target thread, add the target identifier to the metadata of the database message to obtain the extended database message.

[0069] In this embodiment, during the execution of a task request, the target thread can generate a corresponding database message based on the content of the request. The database message consists of three parts: message header, message body, and message trailer. The message body contains core content related to database operations, while the message header and message trailer contain information used to control message transmission and parsing.

[0070] The metadata of the database message is stored in the message header. The metadata adopts a key-value pair storage structure, with each key corresponding to a metadata item. The metadata item includes message length information, message type information, message generation time information, etc. This embodiment can use database protocol message extension technology to transmit the target identifier, effectively solving the identifier maintenance problem in statement caching scenarios. Specifically, for example, by extending the database driver protocol, a dedicated field carrying the target identifier can be added to the database's P message (Parse command message) and Q message (simple query message).

[0071] Adding a target identifier to metadata can begin by first adding a predefined key to the metadata's key-value pair structure. This key must not be duplicated from existing metadata entries, ensuring that existing metadata information is not overwritten. Next, the multiple dimensions corresponding to the target identifier are serialized. This serialization process encodes each dimension into a continuous byte stream according to predefined encoding rules, ensuring that the byte stream is compatible with the metadata's storage structure. Finally, the serialized byte stream is written to the metadata's storage area as the value corresponding to the newly added key.

[0072] For example, the extended message can add an extended identifier field, a target identifier length field, and a target identifier value field to the original structure. These three fields together constitute the extended fields in the metadata, used to store complete information about the target identifier. This scheme transmits the target identifier as protocol layer metadata, avoiding modification of the SQL statement itself. Furthermore, if the target database or intermediate processing node does not support parsing the extended fields, the extended fields can be ignored without affecting the parsing and execution of the original database message.

[0073] After completing the above operations, the resulting extended database message contains both the complete content of the original database message and target identification information for link tracing, which can be identified and parsed by relevant devices during transmission. Upon receiving the extended database message, the database can extract the target identifier through a protocol parser and establish a persistent association with the generated preprocessed handle, ensuring that the tracing context is maintained when executing subsequent operations through this handle.

[0074] S204. Execute the target operation in the target database based on the extended database message.

[0075] In this embodiment, the extended database message can be sent to the target database via a preset transmission link. After receiving the extended database message, the target database can parse the message, for example, verify the integrity of the message, extract metadata information from the message header, and read the core operation content in the message body.

[0076] By parsing the extended identifier field, target identifier length field, and target identifier value field in the extended fields, the complete information of the target identifier can be accurately obtained. During the parsing process, the byte stream data corresponding to the target identifier can be extracted from the metadata, and then the byte stream data can be deserialized to restore the multi-dimensional target identifier. This is then temporarily stored in the temporary storage area of ​​the target database. This temporary storage area is associated with the currently executed database operation and can be accessed at any time during the operation.

[0077] Subsequently, the corresponding target operation can be executed based on the core operation content in the message body. For example, if the task request is a data query request, the target operation could be to retrieve data from the corresponding data table in the database according to the query conditions; if the task request is a data update request, the target operation could be to modify the corresponding record in the data table according to the update conditions.

[0078] During the execution of the target operation, the target identifier in the temporary storage area can be associated with relevant information about the operation execution. This association information includes the start time of the operation execution, the names of the data tables involved in the operation, and the status of the operation execution, ensuring that the corresponding operation process can be traced back through the target identifier later.

[0079] When the target operation fails or an exception occurs, database execution information related to the exception can be recorded based on the target identifier to support fault location and problem backtracking based on the target identifier, thereby improving the efficiency of fault location and problem backtracking.

[0080] The method provided in this application generates a target identifier for a task request based on user-input task request, which is related to the identifier generation time, working node, and task sequence. When the task request enters the service, the target identifier is configured into the target thread executing the task request. When transmitting the database message corresponding to the task request through the target thread, the target identifier is added to the metadata of the database message to obtain an extended database message. The target operation is then executed in the target database based on the extended database message. This achieves effective transmission of the link identifier during the database access phase, closely linking database execution information with the upper-layer request, and achieving end-to-end complete link tracing and accurate performance analysis. Compared with existing technologies, this method overcomes the difficulty of link identifier transmission during the database access phase, ensuring uninterrupted tracing, providing more accurate and comprehensive information for fault location, and providing stronger data support for performance optimization. Furthermore, it avoids modifying SQL statements, reduces the additional performance overhead caused by modifying SQL statements, and avoids situations where some SQL statements cannot be extended for modification, thereby improving the continuity of link tracing and performance monitoring, and the accuracy and effectiveness of fault location and performance optimization.

[0081] The following section provides a detailed explanation of how the target identifier is configured to the target thread that executes the task request in step S202. Figure 3 This is a flowchart illustrating another data processing method provided in an embodiment of this application. Figure 3 As shown, the aforementioned step S202 may specifically include:

[0082] S301. Intercept and obtain the target identifier from the task request.

[0083] One possible implementation is to obtain the target identifier from the task request using an interceptor.

[0084] Specifically, bytecode enhancement technology can be used to intercept Java Database Connectivity (JDBC) methods such as `prepareStatement` and `execute`. The interceptor can first determine the location of the field in the task request that carries the target identifier. This field location is a preset, fixed position associated with the message format of the task request. Specifically, the location of the field corresponding to the target identifier can be obtained by parsing the task request header; for example, it can be obtained by parsing the target identifier from the upstream HTTP header (such as X-Trace-Id).

[0085] Once the field location is determined, the data content in that field can be read. This data content is a multi-dimensional byte stream of data corresponding to the target identifier after serialization, such as the 64-bit target identifier generated by the aforementioned improved snowflake algorithm.

[0086] Another possible implementation is to obtain the target identifier by combining custom annotations with a message middleware mechanism. For example, a custom annotation (such as @TraceIdInject) can be added to the entry method of the task request, and aspect-oriented programming can be used to intercept the method call and extract the target identifier from the request context (such as HTTP headers, message queue headers) or a global identifier generator.

[0087] S302. Configure the target identifier in the context of the target thread.

[0088] In this embodiment, the target thread's context is the core area for storing various related information during thread execution, and the context area is divided into multiple functional sub-regions. Each sub-region corresponds to different types of information storage, including a sub-region specifically for storing tracing identifiers. This sub-region can be implemented, for example, based on the ThreadLocal mechanism, and has a fixed storage address and storage capacity to store the target identifier.

[0089] Specifically, the storage address of the tracing identifier sub-region in the context region can be obtained first. This can be achieved by querying the address mapping table of the thread context. The address mapping table records the address range and corresponding information type of each functional sub-region, where the address of the tracing identifier sub-region is directly associated with the storage address of ThreadLocal. After obtaining the storage address, the target identifier can be written to the sub-region corresponding to that address.

[0090] After successful configuration, when the target thread performs operations such as database message transmission, it can directly read the target identifier from this sub-area of ​​the context without having to obtain it from the task request again, thus improving operational efficiency. For example, by intercepting relevant JDBC methods, the target identifier can be directly extracted from the ThreadLocal context, ensuring the efficiency and accuracy of identifier transmission.

[0091] Optionally, after the task request is processed, the target identifier stored in the target thread context can be cleared to avoid cross-request interference caused by residual target identifiers in thread reuse scenarios.

[0092] The method provided in this application intercepts and obtains the target identifier from the task request, and then configures the target identifier into the context of the target thread. This ensures a one-to-one correspondence between the target identifier and the task request, avoids incorrect identifier configuration, and enables the target thread to quickly obtain the target identifier when performing subsequent operations, thereby improving the efficiency of link tracing and further enhancing the accuracy and continuity of link tracing during the database access phase.

[0093] The following section details how the target identifier is added to the metadata of the database message in step S203 to obtain the extended database message. Step S203 specifically includes: adding the target identifier to an extended field in the metadata of the database message to generate an extended database message, wherein the extended field includes the value of the target identifier.

[0094] Specifically, the metadata of database messages adopts a standardized key-value pair storage structure. Each key-value pair corresponds to a metadata item. The key name of the metadata item is unique, and the naming of the key name follows a preset naming convention to ensure that different metadata items will not be confused. The original fields in the metadata include message length field, message type field, message generation time field, etc., and the key name and data format of each field are clearly defined.

[0095] To add a target identifier, an extended field can be added to the metadata. This extended field is implemented based on database protocol message extension technology. For example, a dedicated field can be added to the P and Q messages of the database. The key name of this extended field is predefined and is specifically used to store the link tracing identifier (i.e., the target identifier). The extended field stores the value of the target identifier, for example, the value of the 64-bit target identifier generated by the aforementioned improved snowflake algorithm.

[0096] When adding an extended field, you can first add a new entry at the end of the metadata key-value pair list. The key of this entry is the preset extended field key name, and the value of the entry is empty or a preset value. Then, write the value of the target identifier into the value field of this entry to generate an extended database message containing extended metadata.

[0097] After receiving the extended database message, the database can extract the target identifier through the protocol parser and establish a persistent association with the generated preprocessed handle, ensuring that the tracking context can still be maintained when executing through the handle later, effectively solving the problem of identifier preservation in statement caching scenarios.

[0098] In one possible implementation, the extended field may also include an extended field identifier field and a target identifier length field.

[0099] The extended field identifier identifies the type and purpose of the extended field. This field uses a preset fixed code, which allows for quick identification of the information type stored in the extended field. When parsing metadata, reading this code is sufficient to determine the purpose of the extended field, eliminating the need for complex parsing of the value field's content.

[0100] The target identifier length field stores the byte length of the target identifier, and its value is the number of bytes in the target identifier. When parsing the extended field, the value of this length field can be read first, and then the corresponding byte length of data can be read from the value field based on this value, avoiding situations where the data length is insufficient or the data is redundant.

[0101] Optionally, the order in which the extended field identifier field and the target identifier length field are added can be placed before the target identifier value field. This way, the receiver can read the identifier field and the length field first during parsing, and then read the target identifier value field based on the information in these two fields, thus improving parsing efficiency.

[0102] For example, the length of the extended field identifier field can be 1 byte, and its value is a fixed value "T"; the length of the target identifier length field can be 2 bytes, and its value is related to the length of the target identifier.

[0103] The method provided in this application provides that by adding an extended field containing a target identifier to the metadata of the database message, the link tracing identifier can be closely associated with the database message. This ensures that the database message always carries link tracing information during transmission and processing, and that the task request corresponding to the message can be accurately identified during the subsequent database execution phase. This achieves end-to-end link tracing, solves the problem of link identifier loss during the database access phase in the prior art, and improves the accuracy of fault location and performance analysis.

[0104] Below, taking the improved snowflake algorithm as an example, we will provide a detailed explanation of how the target identifier of the task request is generated based on the user's input task request in the aforementioned step S201. Figure 4 This is a flowchart illustrating another data processing method provided in an embodiment of this application. Figure 4 As shown, the aforementioned step S201 may specifically include:

[0105] S401. Obtain the worker node identifier and the task sequence identifier corresponding to the task request.

[0106] In this embodiment, the process of obtaining the worker node identifier corresponding to the task request can first determine the processing node where the task request is currently located. The processing nodes are the various functional nodes that the task request passes through in the service process, and each processing node has a unique worker node identifier.

[0107] The worker node identifier can be obtained by querying the configuration information of the processing node. The configuration information of the processing node includes the identity code, function code, and cluster code of the processing node. The identity code of the processing node can be used as the worker node identifier.

[0108] For example, the corresponding worker node identifier can be determined based on the data center, machine, etc., where the processing node is located. For instance, the worker node identifier can consist of a 5-bit data center identifier and a 5-bit machine identifier, such as data center 5 and machine 3, whose corresponding worker node identifier could be 0010100011 (binary). Optionally, the length of the worker node identifier can be 10 bits, supporting a maximum of 1024 worker node identifiers.

[0109] The task sequence identifier of a task request is determined based on the sequence number of concurrent task requests on the processing node at the same time (within one millisecond). For example, assuming the worker node identifier of a task request is 0010100011, the order of the task request in the task sequence can be determined based on the task sequence of concurrent task requests on machine 3 in data center 5 at the same time, and this order can be used as the task sequence identifier of the task request.

[0110] For example, assuming the task sequence identifier is 12 bits long and supports the generation of 4096 task sequence identifiers per node per millisecond, if machine 3 in data center 5 is the 12th task request to generate the target identifier in a certain millisecond of concurrent task requests, then the task sequence identifier of this task request can be 000000001100.

[0111] S402. Generate a target identifier based on the preset symbol bit, work node identifier, task sequence identifier, and current time.

[0112] The preset sign bit is used to ensure that the generated target identifier is a positive integer, such as a specific value (e.g., 0). The current time can be represented by the difference in milliseconds between the current time and the custom epoch time, and its length in the target identifier can be, for example, 41 bits, which can support the generation of timestamps with a length of approximately 69 years.

[0113] In this step, preset symbol bits, work node identifiers, task sequence identifiers, and the current time can be combined according to a preset splicing order. The splicing order can be set according to actual needs. For example, the preset symbol bits can be spliced ​​first, followed by the current time, work node identifier, and task sequence identifier to obtain the corresponding target identifier.

[0114] For example, the structural design of the target identifier can be as shown in Table 1 below:

[0115] Table 1

[0116]

[0117] Optionally, the current time in the target identifier can be generated according to the following rules:

[0118] Rule 1: When the current time is less than the generation time of the previous identifier, and when the current time is greater than or equal to the generation time of the previous identifier, generate the target identifier based on the preset symbol bit, the worker node identifier, the task sequence identifier, and the current time.

[0119] Specifically, the generation time of the previous identifier refers to the time contained in the previously generated target identifier, while the current time refers to the time when the current target identifier is generated. The comparison between the current time and the generation time of the previous identifier can be achieved by comparing the order of the two times in their encoding. The larger the encoding value, the later the time.

[0120] When the chronological code corresponding to the current time is less than the chronological code corresponding to the generation time of the previous identifier, it indicates that the current time is earlier than the generation time of the previous identifier, indicating a clock rollback issue. In this case, a waiting strategy can be adopted until the time corresponding to the current time is greater than or equal to the generation time of the previous identifier, and then the data corresponding to the current time in the target identifier is generated based on the current time.

[0121] The above waiting strategy can effectively handle the clock rollback problem and improve the system's fault tolerance.

[0122] Rule 2: When the current time is equal to the generation time of the previous identifier, the time information in the target identifier is the same as the time information of the previous identifier, and the task sequence identifier in the target identifier is the task sequence identifier of the previous identifier plus a specific value.

[0123] Specifically, "current time equals the generation time of the previous identifier" refers to a scenario where the current time is exactly the same as the time of the previous identifier. In this case, when generating the target identifier, the time is consistent with the time of the previous identifier, i.e., the code corresponding to the current time is used. The task sequence identifier is created by adding a specific value to the sequential code portion of the task sequence identifier of the previous identifier. The specific value is a preset fixed increment, such as adding 1, which allows the adjusted sequential code to be distinguished from the original sequential code.

[0124] After adjusting the sequence encoding, the other dimensions of the task sequence identifier remain unchanged. Then, the adjusted task sequence identifier is concatenated with the preset symbol bit, work node identifier, and time to generate a new target identifier.

[0125] Optionally, when the serial number exceeds the maximum value (e.g., the maximum number of serial numbers supported is 4096, but the serial number is greater than 4095), the current time can be set to the next millisecond, the serial number can be reset to 0, and then the target identifier can be generated.

[0126] Rule 3: When the current time is greater than the generation time of the previous identifier, generate a target identifier based on the preset symbol bit, worker node identifier, task sequence identifier, and current time. The task sequence identifier is the initial value.

[0127] Specifically, "the current time is greater than the generation time of the previous identifier" refers to a scenario where the chronological code corresponding to the current time is greater than the generation time of the previous identifier. In this case, when generating the target identifier, the time corresponding to the current time can be used directly, while the task sequence identifier adopts an initial value, which is a preset default value for the task sequence identifier, such as 0.

[0128] Then, the preset symbol bit, the working node identifier, the task sequence identifier of the initial value, and the current time are concatenated in a preset order to generate the target identifier.

[0129] The method provided in this application obtains the worker node identifier and task sequence identifier corresponding to the task request, generates a target identifier by combining a preset symbol bit and the current time, and adjusts the identifier generation rules according to the relationship between the current time and the generation time of the previous identifier. This ensures that the generated target identifier has uniqueness and validity, enabling the target identifier to accurately associate various key information of the task request, providing a reliable identifier foundation for subsequent link tracing, while avoiding link tracing confusion caused by identifier duplication, and improving the accuracy and reliability of link tracing.

[0130] Specifically, the target identifier is generated by concatenating the preset sign bit, working node identifier, initial value task sequence identifier, and current time in a preset order. For example, it can be generated using the following bitwise operation formula:

[0131]

[0132] Where id is the target identifier. For the current time, For identifying work nodes, Used as a task sequence identifier.

[0133] Optionally, the method may also include adding a target identifier to the response message of the target database. That is, the database may also carry the same target identifier and server timestamp information in the response message, forming a complete bidirectional link tracing mechanism to extend the link tracing identifier to the response stage of the database. This allows the response message and the request message to be associated through the target identifier, realizing complete request-response link tracing, thereby improving the integrity of end-to-end link tracing and providing more comprehensive data support for fault location and performance analysis.

[0134] Specifically, after executing the target operation, the target database generates a corresponding response message. This response message is used to report the result of the operation to the initiator of the task request. The response message also includes a header, body, and trailer. In the header metadata of the response message, an extended field identical to that in the request message can be added. This extended field includes an extended identifier field, a target identifier length field, and a target identifier value field. The target identifier corresponding to the current operation is serialized and written into the value field of this extended field. Simultaneously, the extended identifier field and the target identifier length field are added to ensure that the metadata structure of the response message is consistent with that of the request message.

[0135] During transmission, the response message carries target identification information. After receiving the response message, the receiver can obtain the target identification by parsing the extended fields in the metadata, thereby associating the response result with the original task request and realizing closed-loop tracking of the request and response.

[0136] Optionally, the method may also include storing the SQL statement associated with the target identifier in the database log when executing the SQL statement in the database message in the target database.

[0137] Specifically, when the target database executes the SQL statement in the message body, its log subsystem can capture the target identifier through the protocol extension mechanism, associate it with the executed SQL statement, and record it. The operation log contains information such as the content of the SQL statement, the time of operation execution, and the result of operation execution.

[0138] When storing operation logs, a new field can be added to the log record to store the byte stream data of the corresponding target identifier after serialization. During the storage process, it is ensured that the target identifier corresponds one-to-one with the SQL statement, and the operation log of each SQL statement is associated with a unique target identifier.

[0139] For example, the storage format of database logs can adopt a structured storage method, such as a table-based storage structure, which includes multiple fields such as SQL statement field, execution time field, execution result field, and target identifier field. The fields are interconnected, and the corresponding SQL statement and operation information can be retrieved through the target identifier field.

[0140] Log collection agents deployed on the database side can extract target identifiers from logs using pre-configured parsing rules, and send them as independent fields along with the original log text to central storage such as Elasticsearch. Simultaneously, application service link data is also aggregated to the same storage using the same target identifier as the key.

[0141] This method allows linking the trace identifier to the database's operation logs, enabling subsequent analysis of database operations to trace the corresponding task requests, execution threads, and other information through the target identifier, thereby improving the accuracy and efficiency of fault location.

[0142] Optionally, a join query using the target identifier as the key can be used to associate application nodes with database SQL execution records. The database node is then dynamically rendered in the call chain topology graph, displaying its SQL details, execution time, and original log links, thus constructing a complete observable link from the gateway to the database kernel. Subsequent troubleshooting or performance analysis can then use the target identifier to query the corresponding database operation logs, obtain the execution status of SQL statements, and quickly locate the root cause of the problem, improving the efficiency of problem handling.

[0143] Figure 5 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application. Figure 5 As shown, the device may include: a first processing module 11, a first control module 12, a second control module 13, and a third control module 14.

[0144] The first processing module 11 is used to generate a target identifier for the task request based on the task request input by the user. The target identifier is related to the identifier generation time, working node, and task sequence.

[0145] The first control module 12 is used to configure the target identifier into the target thread that executes the task request when the task request enters the service.

[0146] The second control module 13 is used to add the target identifier to the metadata of the database message when transmitting the database message corresponding to the task request through the target thread, so as to obtain the extended database message.

[0147] The third control module 14 is used to perform target operations in the target database based on the extended database message.

[0148] Optionally, the first control module 12 is specifically used to intercept and obtain the target identifier from the task request. The target identifier is then configured into the context of the target thread.

[0149] Optionally, the second control module 13 is specifically used to add the target identifier to the extended field of the metadata of the database message, and generate an extended database message, wherein the extended field includes the value of the target identifier.

[0150] Optionally, the extended fields may also include an extended field identifier field and a target identifier length field.

[0151] Optionally, the first processing module 11 is specifically used to obtain the worker node identifier corresponding to the task request and the task sequence identifier of the task request. Based on the preset symbol bit, worker node identifier, task sequence identifier, and current time, a target identifier is generated.

[0152] Optionally, the first processing module 11 is specifically used to generate a target identifier based on a preset symbol bit, a work node identifier, a task sequence identifier, and the current time when the current time is less than the generation time of the previous identifier, and when the current time is greater than or equal to the generation time of the previous identifier.

[0153] Optionally, the third control module 14 is also used to add a target identifier to the response message of the target database, and / or, when executing the SQL statement in the database message of the target database, associate the SQL statement with the target identifier and store it in the database log.

[0154] The data processing apparatus provided in this application embodiment can execute the data processing method in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described again here.

[0155] Figure 6 This is a schematic diagram of an electronic device provided in an embodiment of this application. The electronic device is used to execute the aforementioned data processing method. Figure 6 As shown, the electronic device 600 may include at least one processor 601, a memory 602, and a communication interface 603.

[0156] The memory 602 is used to store programs. Specifically, the program may include program code, which includes computer operation instructions.

[0157] The memory 602 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0158] The processor 601 is used to execute computer execution instructions stored in the memory 602 to implement the methods described in the foregoing method embodiments. The processor 601 may be a CPU, an Application Specific Integrated Circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application.

[0159] The processor 601 can communicate and interact with external devices through the communication interface 603. In specific implementations, if the communication interface 603, memory 602, and processor 601 are implemented independently, they can be interconnected via a bus to complete communication. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc., but this does not imply that there is only one bus or one type of bus.

[0160] Optionally, in a specific implementation, if the communication interface 603, memory 602, and processor 601 are integrated on a single chip, then the communication interface 603, memory 602, and processor 601 can communicate through an internal interface.

[0161] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0162] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-described method.

[0163] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0164] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.

[0165] The division of units is merely a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0166] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0167] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0168] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0169] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0170] Finally, it should be noted that other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein, and is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.

Claims

1. A data processing method, characterized in that, include: Based on the task request input by the user, a target identifier for the task request is generated, and the target identifier is related to the identifier generation time, work node, and task sequence. When the task request enters the service, the target identifier is configured in the target thread that executes the task request; When transmitting the database message corresponding to the task request through the target thread, the target identifier is added to the metadata of the database message to obtain an extended database message; The target operation is performed in the target database based on the extended database message.

2. The method according to claim 1, characterized in that, The step of configuring the target identifier into the target thread that executes the task request includes: The target identifier is obtained by intercepting the task request; Configure the target identifier into the context of the target thread.

3. The method according to claim 1, characterized in that, Adding the target identifier to the metadata of the database message to obtain an extended database message includes: The target identifier is added to an extended field in the metadata of the database message to generate the extended database message, wherein the extended field includes the value of the target identifier.

4. The method according to claim 3, characterized in that, The extended fields also include an extended field identifier field and a target identifier length field.

5. The method according to claim 1, characterized in that, The step of generating a target identifier for a task request based on user input includes: Obtain the worker node identifier corresponding to the task request and the task sequence identifier of the task request; The target identifier is generated based on the preset symbol bit, the working node identifier, the task sequence identifier, and the current time.

6. The method according to claim 5, characterized in that, The step of generating the target identifier based on the preset symbol bit, the working node identifier, the task sequence identifier, and the current time includes: When the current time is less than the generation time of the previous identifier, and when the current time is greater than or equal to the generation time of the previous identifier, the target identifier is generated according to the preset symbol bit, the working node identifier, the task sequence identifier, and the current time.

7. The method according to any one of claims 1-6, characterized in that, The method further includes: Add the target identifier to the response message of the target database; And / or, When the target database executes the SQL statement in the database message, the SQL statement is associated with the target identifier and stored in the database log.

8. A data processing apparatus, characterized in that, include: The first processing module is used to generate a target identifier for the task request based on the task request input by the user. The target identifier is related to the identifier generation time, working node, and task sequence. The first control module is used to configure the target identifier into the target thread that executes the task request when the task request enters the service; The second control module is used to add the target identifier to the metadata of the database message when transmitting the database message corresponding to the task request through the target thread, so as to obtain an extended database message. The third control module is used to perform target operations in the target database based on the extended database message.

9. An electronic device, characterized in that, include: Memory, processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the processor to perform the method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1-7.

11. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method described in any one of claims 1-7.

Images