Universal Golang asynchronous refresh buffer frame
By designing a general Golang asynchronous refresh buffer framework, the problem of lack of asynchronous refresh buffer framework suitable for different types of data in the existing technology is solved, and stable large-scale writes and error handling for different types of data is realized, which improves the System stability and reliability.
Patent Information
- Application Number
- CN202411332701.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-24
- Publication Date
- 2025-02-07
AI Technical Summary
The existing open source community lacks a general asynchronous refresh buffer framework suitable for different types of data, and the existing framework cannot effectively handle errors, resulting in data loss or error propagation, affecting system stability and reliability.
A general Golang asynchronous refresh buffer framework is designed, and asynchronous write, automatic refresh and error handling functions are realized by defining the Buffer structure and the Container interface. The framework uses channels and context to pass data, signals and error messages between coroutines to achieve concurrent processing.
It realizes stable large-scale writing of different types of data, provides a modular and scalable buffer design, ensures the consistency and correctness of buffer behavior, avoids data loss and error propagation, and improves the stability of the system. Sex and reliability.
Smart Images
Figure CN119396519A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of software application cache processing, and specifically to a general Golang asynchronous refresh buffer framework. Background Art
[0002] In many practical scenarios, the process of writing data to external storage is a very time-consuming operation. In order to alleviate this problem, the common practice is to maintain a Buffer to aggregate the data and then write it to the external storage. For example, in the scenario where the Golang application client inserts data into Clickhouse (a type of database), the process of writing data to Clickhouse must be batch processed, so asynchronous refresh of the buffer is a must.
[0003] In actual backend development scenarios, in addition to Clickhouse insertion, various data require the ability to asynchronously refresh buffers (such as ck data insertion, kafka message writing, Postgres data insertion, batch reporting of monitoring indicators, etc.); it must be compatible with various types of data, ensure the correctness of various behaviors under various types of data within the framework of asynchronous refresh buffers, and provide space for error handling.
[0004] Since this is a very business-related scenario, there is no good related component library in the existing open source community. There is only an asynchronous refresh buffer framework without generic support that will no longer be maintained after 2020. This framework has no generic support and cannot handle different types of data well. At the same time, various error situations may be encountered during the asynchronous refresh buffer process, such as network failures, data format errors, etc. If the framework cannot provide an effective error handling mechanism, it may cause data loss or error propagation, affecting the stability and reliability of the system. Summary of the invention
[0005] The purpose of this application is to provide a general Golang asynchronous refresh buffer framework to solve the technical problems raised in the above background technology.
[0006] To achieve the above purpose, the present application provides the following technical solutions: a general Golang asynchronous refresh buffer framework,
[0007] Includes data containers for storing different types of data;
[0008] Buffer structure, define the Buffer structure so that it has an asynchronous write function cache mechanism, and can automatically refresh and handle errors. By using different channels and contexts, data, signals and error information can be passed between different coroutines, thereby achieving concurrent processing;
[0009] Define the Container interface for the data container and implement the put, flush, isFull, and reset methods. At the same time, define and implement the Put method, Flush method, Close method, and NewBuffer method for the Buffer structure separately, so that the Buffer structure has the ability to store data, refresh data, close, and create Buffer.
[0010] Preferably, the data container is a Data Container Interface, and the put method of the Container interface is to put a piece of data into the Container. If the return value is an error type, it indicates whether an error occurs when putting the data. If no error occurs, it returns.
[0011] Preferably, the flush method of the Container interface is to refresh the Container and process the refresh data. When flush is implemented as a synchronous refresh, the put method of the Container will not be called, and the Container can clear its data to reset the Container.
[0012] When flush is implemented as an asynchronous refresh, the put method of the data container will still be called when the refresh is executed. The Container should separate the data that needs to be refreshed and processed from it and reset itself;
[0013] If the return value is an error type, it indicates whether an error occurred during the refresh process. If no error occurred, it is returned.
[0014] Preferably, when the flush method is called, the Container should be reset.
[0015] Preferably, the isFull method of the Container interface is used to determine whether the Container is full, and the return value is a Boolean value, which returns true if the container is full, otherwise returns false;
[0016] The reset method of the Container interface is used to be called when the flush method returns an error to reset the data container. This method has no parameters and return values and is used to restore the container to its initial state or clear the data when the refresh operation fails.
[0017] Preferably, the definition of the Buffer structure includes
[0018] Container, a container that implements the Container[T] interface, used to store cached data;
[0019] context and cancel, the context of the cache close signal and the function that can be called to cancel the context, used to send a signal to close the cache;
[0020] dataChan, used to asynchronously put data into the Container's channel to achieve lock-free and non-blocking writing;
[0021] flushSignalChan, a channel for transmitting flush signals;
[0022] cleanedSignalChan, a channel for transmitting buffer cleanup completion signals;
[0023] errChan, a channel for sending Errors to Buffer users;
[0024] Config, Buffer configuration, including the following configuration items:
[0025] chanBufSize int: dataChan size, which means the buffer will block writing after holding unread data of this size;
[0026] flushInterval time.Duration: indicates the cycle of automatically triggering flush;
[0027] logger*logr.Logger: This logger is used for log output in Buffer.
[0028] Preferably, the Buffer structure implements the Put method to check whether the Buffer has been closed, specifically, to check whether the channel returned by buffer.context.Done() has been closed. If it has been closed, ErrClosed is directly returned.
[0029] Preferably, the Flush method implemented for the Buffer structure is to check whether the Buffer has been closed, specifically, to check whether the channel returned by buffer.context.Done() has been closed. If it has been closed, ErrClosed is directly returned. For the scenario of manually triggering flush, since the flush signal channel buffer.flushSignalChan is designed in the structure, it is only necessary to directly pass the semaphore to buffer.flushSignalChan.
[0030] Preferably, the Buffer structure implements the Close method comprising the following steps:
[0031] S1. First check whether the Buffer has been closed. If it has been closed, no other operations are required and the ErrClosed error is returned directly;
[0032] S2. Call the cancel function to prevent further Put, Flush, and Close method calls;
[0033] S3. Wait for buffer.cleanedSignalChan to close, that is, after the buffer data is cleaned, close each semaphore channel.
[0034] Preferably, the Buffer structure implements the NewBuffer method to check the correctness of Config and assign default values, uses WithCancel to initialize Context, obtains the cancel function, completes the initialization of Buffer, and then calls buffer.run to formally activate the buffer; in the run coroutine, manages the life cycle of the data in the buffer, processes each semaphore of the buffer, starts an infinite loop, and uses Golang's select to monitor 4 channels:
[0035] buffer.dataChan: listens to the data passed in by buffer.Put, writes it to the Container through putAndCheck() and calls container.isFull to check whether the Container is full. If it is full, call container.flush to refresh the buffer. If an error occurs during put / flush, call container.reset to reset the container.
[0036] ticker.C: The ticker periodically sends signals to automatically refresh the buffer and also handle errors;
[0037] buffer.context.Done(): This channel indicates that the buffer has been explicitly closed, so it is cleared here;
[0038] buffer.flushSignalChan: Manually flush the buffer because the timer failed to complete its task, flush the buffer immediately when the timer times out.
[0039] Compared with the prior art, the beneficial effects of this application are:
[0040] This application uses the characteristics of channel to handle concurrent access to Buffer and implement lock-free Buffer. It can also use channel to pass semaphores to control the life cycle of Buffer (creation, readiness, closure, completion and cleanup), so that developers can use this asynchronous refresh buffer framework to implement stable large-scale data writing to various data sources. This design makes the Buffer structure more modular and extensible, and can adapt to different application scenarios and requirements. At the same time, through the definition and use of interfaces, we can ensure the consistency and correctness of the behavior of Buffer instances and avoid potential errors and problems. BRIEF DESCRIPTION OF THE DRAWINGS
[0041] Figure 1 This is the framework diagram of the application;
[0042] Figure 2 Build a flow chart for this application framework. DETAILED DESCRIPTION
[0043] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0044] In the description of the present application, it should be understood that the terms "upper", "lower", "front", "back", "left", "right", "top", "bottom", "inside", "outside", etc., indicating orientations or positional relationships, are based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present application.
[0045] Example:
[0046] See also Figure 1 and Figure 2 , this application provides a technical solution:
[0047] A general Golang asynchronous flush buffer framework.
[0048] Includes data containers for storing different types of data;
[0049] Buffer structure, define the Buffer structure so that it has an asynchronous write function cache mechanism, and can automatically refresh and handle errors. By using different channels and contexts, data, signals and error information can be passed between different coroutines, thereby achieving efficient concurrent processing;
[0050] Define the Container interface for the data container and implement the put, flush, isFull, and reset methods. At the same time, define and implement the Put method, Flush method, Close method, and NewBuffer method for the Buffer structure separately, so that the Buffer structure has the ability to store data, refresh data, close, and create Buffer.
[0051] The framework building method includes the following steps:
[0052] Step 1: Abstract the data container in the Buffer as Data Container Interface, requiring the implementation of put, flush, isFull, and reset methods to prepare for subsequent Buffer-related implementations;
[0053] The specific implementation method is to create an abstract Container interface and define four methods: put, flush, isFull, and reset. These methods together define an abstract data container interface that can be used for different types of data storage and processing and is compatible with a variety of data structures:
[0054] put(data T)error: puts a piece of data into the Container. If the return value is an error type, it indicates whether an error occurred when putting the data. If no error occurred, it returns;
[0055] flush()error: flushes the Container and processes the refreshed data; when this method is called, the Container should be reset. When flush() is implemented as a synchronous refresh, the put method of the Container will not be called, so the Container can clear its data to reset the Container. When flush() is implemented as an asynchronous refresh, the put method of the container will still be called when the refresh is performed, so the Container should separate the data that needs to be refreshed and processed from it and reset itself. If the return value is an error type, it indicates whether an error occurred during the refresh process. If no error occurred, it is returned; when the flush method is called, the Container should be reset;
[0056] isFull()bool: Determines whether the Container is full. The return value is a Boolean value. If the container is full, it returns true, otherwise it returns false.
[0057] reset(): Called when flush() returns an error to reset the data container. This method has no parameters and return values. It is used to restore the container to its initial state or clear the data when the flush operation fails.
[0058] Step 2, define a Buffer structure, including the Data ContainerInterface, related Config, context Context and corresponding CancelFunc defined in the first step, and various Channels for transmitting Data and semaphores (in order to coordinate and standardize the specific behavior of the container based on the container interface, the design purpose of this Buffer structure is to provide a cache mechanism with asynchronous write function, and support automatic refresh and error handling. By using different channels and contexts, it can pass data, signals and error information between different coroutines, thereby achieving efficient concurrent processing, so as to realize the related functions of asynchronous refresh buffer);
[0059] The specific implementation method is to create a Buffer structure, which contains the following fields:
[0060] Container, a container that implements the Container[T] interface, used to store cached data;
[0061] context and cancel, the context of the cache close signal and the function that can be called to cancel the context, used to send a signal to close the cache;
[0062] dataChan, used to asynchronously put data into the Container's channel to achieve lock-free and non-blocking writing;
[0063] flushSignalChan, a channel for transmitting flush signals;
[0064] cleanedSignalChan, a channel for transmitting buffer cleanup completion signals;
[0065] errChan, a channel for sending Errors to Buffer users;
[0066] Config, Buffer configuration, including the following configuration items:
[0067] chanBufSize int: dataChan size, which means that the buffer will block writing after holding unread data of this size (chanBufSize is an integer variable, which represents the buffer size of the dataChan channel; when the unread data in the buffer of the dataChan channel reaches the size specified by chanBufSize, the coroutine or thread that attempts to write new data to the channel will be blocked until data is read in the buffer and space is released. In order to avoid unexpected memory allocation, the cache size is fixed, and writing is blocked after the buffer is full (golang's channel blocking writing does not cause failure, but delay));
[0068] flushInterval time.Duration: indicates the cycle of automatically triggering flush;
[0069] logger*logr.Logger: This logger is used for log output in Buffer;
[0070] Step 3, implement the Put method for Buffer, which receives data and returns directly without blocking the main process (the Buffer.Put method can be implemented by checking the status of buffer.context.Done() and using an unbuffered channel to send data to ensure that data is added to the buffer under the correct conditions);
[0071] The specific implementation method is to check whether the Buffer has been closed (the closed method used to detect whether the Buffer has been closed (using the select statement to non-blockingly try to read the channel returned by the context.Done() of the buffer. If the context has been closed, take the first branch; if it has not been closed, take the second branch)). Specifically, it checks whether the channel returned by buffer.context.Done() has been closed. If it has been closed, it directly returns ErrClosed; and due to the existence of buffer.dataChan, for Buffer.Put, there is no need to worry about competition and deadlock problems, and data can be directly passed into buffer.dataChan;
[0072] Step 4, implement the Flush method for Buffer and actively trigger the Buffer refresh (by checking the status of buffer.context.Done() and using buffer.flushSignalChan to send the flush signal, you can implement the Buffer.Flush method to ensure that the flush operation is triggered under the correct conditions);
[0073] The specific implementation method is to check whether the Buffer has been closed. Specifically, it is to check whether the channel returned by buffer.context.Done() has been closed. If it has been closed, ErrClosed is returned directly. For the scenario of manually triggering flush, since the flush signal channel buffer.flushSignalChan is designed in the structure, you only need to directly pass the semaphore to buffer.flushSignalChan.
[0074] Step 5, implement the Close method for Buffer, so that Buffer stops receiving data and closes all related coroutines and channels; in the Close method, we need to handle buffer data cleanup and the closing of each semaphore channel in the buffer to avoid channel leakage (the Buffer.Close method can be implemented by checking the status of buffer.context.Done(), calling the cancel function, and waiting for data cleanup to be completed and closing each semaphore channel to ensure that the cache is closed and related resources are released under the correct conditions);
[0075] The specific implementation method is:
[0076] S1. First check whether the Buffer has been closed. If it has been closed, no other operations are required and the ErrClosed error is returned directly;
[0077] S2. Call the cancel function to prevent further Put, Flush, and Close method calls;
[0078] S3, wait for buffer.cleanedSignalChan to close, that is, after the buffer data is cleaned up, close each semaphore channel;
[0079] Step 6, implement the NewBuffer function to initialize the Buffer, and start the data processing coroutine while creating the Buffer (by verifying the configuration, initializing the context, creating a Buffer instance and activating it, the NewBuffer function can be implemented to create and start a cache mechanism with specific functions);
[0080] The specific implementation method is: first check the correctness of Config and assign default values, use WithCancel to initialize Context, get the cancel function, complete the initialization of Buffer, and then call buffer.run to formally activate the buffer;
[0081] In the run coroutine, manage the life cycle of the data in the buffer, process the various semaphores of the buffer, start an infinite loop, and use Golang's select to monitor four channels:
[0082] buffer.dataChan: listens to the data passed in by buffer.Put, writes it to the Container through putAndCheck() and calls container.isFull to check whether the Container is full. If it is full, call container.flush to refresh the buffer. If an error occurs during put / flush, call container.reset to reset the container.
[0083] ticker.C: The ticker periodically sends signals to automatically refresh the buffer and also handle errors;
[0084] buffer.context.Done(): This channel indicates that the buffer has been explicitly closed, so it is cleared here;
[0085] buffer.flushSignalChan: Manually flush the buffer because the timer failed to complete its task, flush the buffer immediately when the timer times out.
[0086] The above shows and describes the basic principles and main features of the present application and the advantages of the present application. For those skilled in the art, it is obvious that the present application is not limited to the details of the above exemplary embodiments, and the present application can be implemented in other specific forms without departing from the spirit or basic features of the present application; therefore, no matter from which point of view, the embodiments should be regarded as exemplary and non-restrictive. The scope of the present application is limited by the attached claims rather than the above description, and it is intended to include all changes within the meaning and scope of the equivalent elements of the claims in the present application, and any figure marks in the claims should not be regarded as limiting the claims involved.
[0087] Although the embodiments of the present application have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present application, and that the scope of the present application is defined by the appended claims and their equivalents.
Claims
1. A general Golang asynchronous refresh buffer framework, characterized by: Includes data containers for storing different types of data; Buffer structure, define the Buffer structure so that it has an asynchronous write function cache mechanism, and can automatically refresh and handle errors. By using different channels and contexts, data, signals and error information can be passed between different coroutines, thereby achieving concurrent processing; Define the Container interface for the data container and implement the put, flush, isFull, and reset methods. At the same time, define and implement the Put method, Flush method, Close method, and NewBuffer method for the Buffer structure separately, so that the Buffer structure has the ability to store data, refresh data, close, and create Buffer.
2. A general Golang asynchronous refresh buffer framework according to claim 1, characterized in that: The data container is a Data Container Interface. The put method of the Container interface is to put a piece of data into the Container. If the return value is an error type, it indicates whether an error occurs when putting the data. If no error occurs, it returns.
3. A general Golang asynchronous refresh buffer framework according to claim 1, characterized in that: The flush method of the Container interface is used to refresh the Container and process the refresh data. When flush is implemented as a synchronous refresh, the put method of the Container will not be called. The Container can clear its data to reset the Container. When flush is implemented as an asynchronous refresh, the put method of the data container will still be called when the refresh is executed. The Container should separate the data that needs to be refreshed and processed from it and reset itself; If the return value is an error type, it indicates whether an error occurred during the refresh process. If no error occurred, it is returned.
4. A general Golang asynchronous refresh buffer framework according to claim 3, characterized in that: When the flush method is called, the Container should be reset.
5. A general Golang asynchronous refresh buffer framework according to claim 4, characterized in that: The isFull method of the Container interface is used to determine whether the Container is full. The return value is a Boolean value. If the container is full, it returns true, otherwise it returns false. The reset method of the Container interface is used to be called when the flush method returns an error to reset the data container. This method has no parameters and return values and is used to restore the container to its initial state or clear the data when the refresh operation fails.
6. A general Golang asynchronous refresh buffer framework according to claim 5, characterized in that: The definition of the Buffer structure includes Container, a container that implements the Container[T] interface, used to store cached data; context and cancel, the context of the cache close signal and the function that can be called to cancel the context, used to send a signal to close the cache; dataChan, used to asynchronously put data into the Container's channel to achieve lock-free and non-blocking writing; flushSignalChan, a channel for transmitting flush signals; cleanedSignalChan, a channel for transmitting buffer cleanup completion signals; errChan, a channel for sending Errors to Buffer users; Config, Buffer configuration, including the following configuration items: chanBufSize int: dataChan size, which means the buffer will block writing after holding unread data of this size; flushInterval time.Duration: indicates the cycle of automatically triggering flush; logger*logr.Logger: This logger is used for log output in Buffer.
7. A general Golang asynchronous refresh buffer framework according to claim 6, characterized in that: The Put method implemented for the Buffer structure is to check whether the Buffer has been closed. Specifically, it checks whether the channel returned by buffer.context.Done() has been closed. If it has been closed, ErrClosed is directly returned.
8. A general Golang asynchronous refresh buffer framework according to claim 7, characterized in that: The Flush method implemented for the Buffer structure is to check whether the Buffer has been closed. Specifically, it checks whether the channel returned by buffer.context.Done() has been closed. If it has been closed, ErrClosed is returned directly. For the scenario of manually triggering flush, since the flush signal channel buffer.flushSignalChan is designed in the structure, it is only necessary to directly pass the semaphore to buffer.flushSignalChan.
9. A general Golang asynchronous refresh buffer framework according to claim 8, characterized in that: The Buffer structure implements the Close method including the following steps: S1. First check whether the Buffer has been closed. If it has been closed, no other operations are required and the ErrClosed error is returned directly; S2. Call the cancel function to prevent further Put, Flush, and Close method calls; S3. Wait for buffer.cleanedSignalChan to close, that is, after the buffer data is cleaned, close each semaphore channel.
10. A general Golang asynchronous refresh buffer framework according to claim 9, characterized in that: The Buffer structure implements the NewBuffer method to check the correctness of Config and assign default values. It uses WithCancel to initialize Context, obtains the cancel function, completes the initialization of Buffer, and then calls buffer.run to formally activate the buffer. In the run coroutine, it manages the life cycle of the data in the buffer, processes the various semaphores of the buffer, starts an infinite loop, and uses Golang's select to monitor 4 channels: buffer.dataChan: listens to the data passed in by buffer.Put, writes it to the Container through putAndCheck() and calls container.isFull to check whether the Container is full. If it is full, call container.flush to refresh the buffer. If an error occurs during put / flush, call container.reset to reset the container. ticker.C: The ticker periodically sends signals to automatically refresh the buffer and also handle errors; buffer.context.Done(): This channel indicates that the buffer has been explicitly closed, so it is cleared here; buffer.flushSignalChan: Manually flush the buffer because the timer failed to complete its task, flush the buffer immediately when the timer times out.