An optimization method and system for link data storage
By using message middleware and caching mechanisms in a microservice architecture, and by delaying consumption and writing to the database uniformly, the problem of low data storage efficiency in distributed tracing is solved, database write efficiency is improved, and resource consumption is reduced.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- FUJIAN TQ ONLINE INTERACTIVE INC
- Filing Date
- 2026-04-15
- Publication Date
- 2026-07-10
AI Technical Summary
In a microservice architecture, distributed tracing has low data storage efficiency, resulting in low database write efficiency and excessive resource consumption.
A message middleware is used to store distributed tracing request data. When the last sub-request is received, an end flag is added, message consumption is delayed, and data is preprocessed in the cache until it is complete and then written to the database.
It improves database write efficiency, reduces the number of database operations, and lowers resource usage time.
Smart Images
Figure CN122364281A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer communication technology, and in particular to an optimization method and system for link data storage. Background Technology
[0002] Currently, to meet the demands of massive data requests, server-side interfaces need to adopt a distributed deployment approach, making microservice architecture the preferred method for large-scale systems. The introduction of microservice architecture necessitates increased investment in operations and maintenance, with distributed trace tracing being an essential tool. The main idea of distributed trace tracing is that when a complete external request arrives, it is broken down into dozens or hundreds of sub-requests within the microservice. All sub-requests carry the same traceId, which identifies whether they belong to the same external request. This data is stored in the database, linked by traceId. However, in current storage, one traceId corresponds to dozens or hundreds of sub-requests, and these requests have hierarchical relationships, leading to inefficiency in data ingestion. For example, after an external request is written to the database, subsequent sub-requests need to retrieve the record based on the traceId, assemble the latest data, and update the database again. This significantly reduces database write efficiency and consumes substantial database resources. Summary of the Invention
[0003] To overcome the above problems, the purpose of this invention is to provide an optimized method and system for link data storage, which can improve the write efficiency of the database and reduce the time that the database occupies a large amount of resources.
[0004] This invention is implemented using the following scheme: an optimization method for link data storage, the method comprising the following steps: Step S1: Set up a message middleware. When distributed tracing generates request data, write all call messages into the message middleware. Step S2: When the message middleware receives the last sub-request, add an end flag; Step S3: After a certain delay, consume messages from the message middleware and determine whether all sub-request data for a request has been fully obtained. If yes, store all of them in the database at once; otherwise, store them in the cache and wait for all sub-request data to arrive before storing them in the database. Step S4: When it is necessary to read link data, first query the data in the cache based on the traceId. If the data does not exist in the cache, then query the data in the database.
[0005] Furthermore, the delay in consuming messages in the message middleware during step S3 is further defined as follows: when a sub-microservice call is triggered, the call messages in the message middleware are consumed after the preset delay time has elapsed.
[0006] Furthermore, the method for determining whether all sub-request data of a request has been completely obtained in step S3 is as follows: when the message middleware receives the last sub-request with an end marker among the sub-requests carrying the same traceId, it indicates that the cached data carrying the same traceId has been complete.
[0007] Furthermore, the step S3 of writing data to the cache and waiting for all sub-request data to arrive before writing it to the database is further described as follows: First, check if there is a record in the cache for the traceId carried by this sub-request. If there is no record, add it to the cache. If there is a record, write this information to the cache and update it. When the last sub-request with the same traceId and the end marker is received, the entire set of sub-requests with the same traceId in the cache can be read out and written to the database.
[0008] Furthermore, the format of the write cache in step S3 is map format, where the key is traceId, the field is the ID of each microservice, and the value is the specific message content.
[0009] An optimized system for link data storage includes a message middleware writing module, an end-of-message flag adding module, a delayed consumption module, and a reading module. The message middleware writing module: sets up a message middleware, and when the distributed tracing generates request data, writes all the call messages into the message middleware; The end flag adding module adds an end flag when the message middleware receives the last sub-request. The delayed consumption module consumes messages from the message middleware after a certain delay, and determines whether all sub-request data of a request has been fully obtained. If so, all of them are stored in the database at once; otherwise, they are written to the cache and written to the database after all sub-request data has been obtained. The reading module, when it needs to read link data, first queries the data in the cache based on the traceId. If the data is not found in the cache, it then queries the data in the database.
[0010] Furthermore, the message consumption in the message middleware in the delayed consumption module is further defined as follows: when a sub-microservice call is triggered after a preset delay time, the call message in the message middleware is consumed after the preset delay time has elapsed.
[0011] Furthermore, the method for determining whether all sub-request data of a request has been completely obtained in the delayed consumption module is as follows: when the message middleware receives the last sub-request with an end marker among sub-requests carrying the same traceId, it indicates that the cached data carrying the same traceId has been complete.
[0012] Furthermore, the process of writing data to the cache in the delayed consumption module, and then writing it to the database after all sub-request data has arrived, is further as follows: First, check if there is a record in the cache for the traceId carried by this sub-request. If there is no record, add it to the cache. If there is a record, write this information to the cache and update it. When the last sub-request with the same traceId and an end marker is received, the entire set of sub-requests with the same traceId in the cache can be read out and written to the database.
[0013] Furthermore, the write cache in the delayed consumption module is in map format, with the key being traceId, the field being the ID of each microservice, and the value being the specific message content.
[0014] The beneficial effects of this invention are as follows: By delaying message consumption, all call messages are written to a message middleware. Messages in the middleware are consumed after a certain delay. If all sub-request data for a request has been fully retrieved, it is all written to the database at once; otherwise, it is written to a cache, and the database is processed uniformly after all sub-request data has arrived. Compared with existing technologies, this invention caches all microservice information for the same request, and only writes it to the database once the request is completed. This invention requires only one database entry, significantly reducing the number of database operations, greatly improving database write efficiency, and reducing the time a large amount of database resources are occupied. Attached Figure Description
[0015] Figure 1 This is a schematic diagram of the method flow of the present invention.
[0016] Figure 2 This is a system principle block diagram of the present invention. Detailed Implementation
[0017] The invention will now be further described with reference to the accompanying drawings.
[0018] Please see Figure 1 As shown, the present invention provides an optimization method for link data storage, the method comprising the following steps: Step S1: Set up a message middleware. When distributed tracing generates request data, write all call messages into the message middleware. Step S2: When the message middleware receives the last sub-request, add an end flag; Step S3: After a certain delay, consume messages from the message middleware and determine whether all sub-request data for a request has been fully obtained. If yes, store all of them in the database at once; otherwise, store them in the cache and wait for all sub-request data to arrive before storing them in the database. Step S4: When it is necessary to read link data, first query the data in the cache based on the traceId. If the data does not exist in the cache, then query the data in the database.
[0019] The present invention will be further described below with reference to a specific embodiment: An optimization method for link data storage. 1. Set up a message middleware. When distributed tracing generates request data, write all call messages into the message middleware. Common message middleware includes RabbitMQ, Kafka, RocketMQ, etc. Message middleware is a commonly used module decoupling component.
[0020] 2. When the message middleware receives the last sub-request, add an end marker; each message has its own message body, and an end marker is added to the message body.
[0021] 3. After a certain delay, consume messages from the message middleware and determine whether all sub-request data for a request has been fully retrieved. If so, store all of them in the database at once; otherwise, store them in the cache and wait for all sub-request data to arrive before storing them in the database. The delay in consuming messages from the message middleware mentioned in section 3.1 is further defined as follows: After a preset delay, when a sub-microservice is triggered, the message consumption within the message middleware begins after the preset delay. Here, a sub-microservice refers to the various sub-services within a specific product or application.
[0022] The method for determining whether all sub-request data of a request has been completely obtained in section 3.2 is as follows: when the message middleware receives the last sub-request with an end marker that carries the same traceId, it indicates that the cached data carrying the same traceId has been completed.
[0023] The process described in 3.3 involves writing data to the cache and then writing it to the database after all sub-request data has been received. Further steps include: first, checking the cache for a record of the traceId carried by the sub-request; if no record is found, adding it to the cache; if a record is found, writing this information to the cache and updating it; and then, when the last sub-request with the same traceId and an end marker is received, all sub-requests with the same traceId can be read from the cache and written to the database.
[0024] The write cache format in 3.3 is a map format, where the key is traceId, the field is the ID of each microservice, and the value is the specific message content.
[0025] For example, 3.1: A one-minute delay is preset for message consumption. After one minute, message consumption begins. A whole batch of messages is fetched for processing at a time. If all messages with a specific traceId in that batch have been processed and assembled (if any message in the batch has an ending status, it means the batch has been fully retrieved), then that batch of messages can be directly written to the database. Otherwise, if the batch is not yet complete, it is first written to the cache, and only after the entire batch is complete is it written to the database.
[0026] For example, 3.2: Suppose an external request comes in and the internal system accesses 5 microservices. In the old approach, the first microservice would send a message to the message middleware. The consumer program would consume the message, query the database at least once (it would be empty at this point), and then write to the database once. The second microservice would also send a message and operate on the database twice as usual. And so on, with 5 microservices requiring 10 database operations.
[0027] In this invention, the external request generates a traceId, which is then carried over to subsequent sub-microservice requests. Each microservice adds this traceId to its message body and sends it to the message middleware.
[0028] When a consumer consumes a message from the message middleware, it first checks the cache center to see if there is a record for this traceId. If there is no record, it adds it to the cache. If there is a record, it writes its own information to the cache and updates it.
[0029] When the traceId reaches the last microservice, the message body of the last message will contain an "end" marker. At this point, all cached records for that traceId can be read, and the data can be assembled using the program's business logic to generate the final data and perform the data insertion operation. This requires only one data insertion operation, instead of the previous ten.
[0030] 4. When it is necessary to read link data, first query the data in the cache based on the traceId. If the data is not found in the cache, then query the data in the database.
[0031] Please see Figure 2 As shown, the present invention also provides an optimized system for link data storage, the system comprising a message middleware writing module, an end-of-message flag adding module, a delayed consumption module, and a reading module. The message middleware writing module: sets up a message middleware, and when the distributed tracing generates request data, writes all the call messages into the message middleware; The end flag adding module adds an end flag when the message middleware receives the last sub-request. The delayed consumption module consumes messages from the message middleware after a certain delay, and determines whether all sub-request data of a request has been fully obtained. If so, all of them are stored in the database at once; otherwise, they are written to the cache and written to the database after all sub-request data has been obtained. The delayed consumption module further defines the consumption of messages in the message middleware by delaying the consumption of messages for a certain period of time: when a sub-microservice call is triggered, the call messages in the message middleware are consumed after the preset delay time has elapsed.
[0032] The method for determining whether all sub-request data of a request has been completely obtained in the delayed consumption module is as follows: when the message middleware receives the last sub-request with an end marker that carries the same traceId, it indicates that the cached data carrying the same traceId has been completed.
[0033] The process of writing data to the cache in the delayed consumption module, and then writing it to the database after all sub-request data has arrived, further involves: first, checking if there is a record in the cache for the traceId carried by the sub-request; if there is no record, adding it to the cache; if there is a record, writing this information to the cache and updating it; and when the last sub-request with the same traceId and an end marker is received, all sub-requests with the same traceId can be read from the cache and written to the database.
[0034] The write cache in the delayed consumption module is in map format, with the key being traceId, the field being the ID of each microservice, and the value being the specific message content.
[0035] The reading module, when it needs to read link data, first queries the data in the cache based on the traceId. If the data is not found in the cache, it then queries the data in the database.
[0036] The above description is only a preferred embodiment of the present invention. All equivalent changes and modifications made within the scope of the claims of the present invention should be included in the scope of the present invention.
Claims
1. An optimization method for link data storage, characterized in that: The method includes the following steps: Step S1: Set up a message middleware. When distributed tracing generates request data, write all call messages into the message middleware. Step S2: When the message middleware receives the last sub-request, add an end flag; Step S3: After a certain delay, consume messages from the message middleware and determine whether all sub-request data for a request has been fully obtained. If yes, store all of them in the database at once; otherwise, store them in the cache and wait for all sub-request data to arrive before storing them in the database. Step S4: When it is necessary to read link data, first query the data in the cache based on the traceId. If the data does not exist in the cache, then query the data in the database.
2. The method for optimizing link data storage according to claim 1, characterized in that: The delay in consuming messages in the message middleware during step S3 is further defined as follows: when a sub-microservice call is triggered, the call messages in the message middleware are consumed after the preset delay time has elapsed.
3. The method for optimizing link data storage according to claim 1, characterized in that: The method for determining whether all sub-request data of a request has been completely obtained in step S3 is as follows: when the message middleware receives the last sub-request with an end marker that carries the same traceId, it indicates that the cached data carrying the same traceId has been completed.
4. The method for optimizing link data storage according to claim 1, characterized in that: The step S3, where data is written to the cache and then uniformly written to the database after all sub-request data has arrived, further involves: first, checking if there is a record in the cache for the traceId carried by this sub-request; if there is no record, adding it to the cache; if there is a record, writing this information to the cache and updating it; and when the last sub-request with the same traceId and an end marker is received, all sub-requests with the same traceId in the cache can be read out and uniformly written to the database.
5. The method for optimizing link data storage according to claim 1, characterized in that: The format for writing to the cache in step S3 is a map format, where the key is traceId, the field is the ID of each microservice, and the value is the specific message content.
6. An optimized system for link data storage, characterized in that: The system includes a message middleware writing module, an end-of-message flag adding module, a delayed consumption module, and a reading module. The message middleware writing module: sets up a message middleware, and when the distributed tracing generates request data, writes all the call messages into the message middleware; The end flag adding module adds an end flag when the message middleware receives the last sub-request. The delayed consumption module consumes messages from the message middleware after a certain delay, and determines whether all sub-request data of a request has been fully obtained. If so, all of them are stored in the database at once; otherwise, they are written to the cache and written to the database after all sub-request data has been obtained. The reading module: when it needs to read link data, it first queries the data in the cache based on the traceId. If the data is not found in the cache, it then queries the data in the database.
7. The link data storage optimization system according to claim 6, characterized in that: The delayed consumption module further defines the consumption of messages in the message middleware by delaying the consumption of messages for a certain period of time: when a sub-microservice call is triggered, the call messages in the message middleware are consumed after the preset delay time has elapsed.
8. The optimized system for link data storage according to claim 6, characterized in that: The method for determining whether all sub-request data of a request has been completely obtained in the delayed consumption module is as follows: when the message middleware receives the last sub-request with an end marker that carries the same traceId, it indicates that the cached data carrying the same traceId has been completed.
9. The link data storage optimization system according to claim 6, characterized in that: The process of writing data to the cache in the delayed consumption module, and then writing it to the database after all sub-request data has arrived, further involves: first, checking if there is a record in the cache for the traceId carried by the sub-request; if there is no record, adding it to the cache; if there is a record, writing this information to the cache and updating it; and when the last sub-request with the same traceId and an end marker is received, all sub-requests with the same traceId can be read from the cache and written to the database.
10. An optimized system for link data storage according to claim 6, characterized in that: The write cache in the delayed consumption module is in map format, with the key being traceId, the field being the ID of each microservice, and the value being the specific message content.