Offset management method and device, electronic equipment, medium and computer product
By identifying the target application's submitted offsets, obtaining the offsets directly or indirectly and storing them into the database, the limitations of the offset management method in the prior art are solved, and unified management and tracking of applications that store offsets in dependency status or file backend are realized.
Patent Information
- Application Number
- CN202510524648.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-04-24
- Publication Date
- 2025-08-08
AI Technical Summary
The offset management method in the prior art is only applicable to applications that actively submit offsets, and cannot effectively manage applications that depend on state or file backend storage offsets, increasing management and maintenance costs.
By identifying the target application's submission offset, the Kafka interface is used to read the offset directly, or indirectly read and parse the redirected files from external storage, obtain the offset and store it in the database, realizing unified management and analysis of the offset data.
It realizes effective management of applications that depend on state or file backend storage offsets, reduces management and maintenance costs, and facilitates downstream applications to track overall consumption.
Smart Images
Figure CN120448362A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of big data technology, and in particular to an offset management method, device, electronic device, medium, and computer product. Background Art
[0002] In the context of big data, Kafka organizes messages through topics and partitions, allowing multiple consumer groups to simultaneously subscribe to different partitions of the same topic, thereby achieving high throughput and scalability. Due to Kafka's inherent design, its internally stored offsets are "untraceable" to external systems, resulting in downstream applications and businesses being unable to determine where the program has consumed or processed messages. Therefore, it is necessary to manage the offsets of Kafka consumers to facilitate downstream tracking.
[0003] The offset management method in related technologies is mainly optimized for applications that actively submit offsets to Kafka. However, for applications that rely on state or file backends (such as HDFS, local file systems, etc.) to manage offsets, it is impossible to manage and synchronize these scattered offset information, which increases the management and maintenance costs. Summary of the Invention
[0004] The present application provides an offset management method, device, electronic device, medium and computer product to solve the problem that the offset management method in the related art is only applicable to applications that actively submit offsets.
[0005] The first aspect of the present application provides an offset management method, including the following steps: identifying the target method for a target application to submit an offset; obtaining the offset of a consumer group corresponding to the target application according to the target method, and analyzing the consumption status of the consumer group corresponding to the target application; and storing the offset and consumption status in a preset database.
[0006] Optionally, the target mode includes a direct mode and an indirect mode, and the offset of the consumer group corresponding to the target application is obtained according to the target mode, including: if the target mode is a direct mode, the offset of the consumer group corresponding to the target application is read based on the target interface; if the target mode is an indirect mode, the transit file list is read, and the consumer group corresponding to the target application and the offset corresponding to the consumer group in the transit file list are extracted.
[0007] Optionally, reading the transit file list includes: determining a target directory path for storing the offset; and reading the transit file list based on the target directory path.
[0008] Optionally, extracting the consumer group corresponding to the target application and the offset corresponding to the consumer group in the transit file list includes: traversing the transit file list to read the file content of each file; parsing the file content according to the file format of each file to obtain key fields, wherein the key fields include a consumer group identifier, a topic name, a partition number, and an offset; matching the consumer group related to the target application and the offset corresponding to the consumer group according to the identification information of the target application.
[0009] Optionally, analyzing the consumption status of the consumer group corresponding to the target application includes: checking the transfer file list; if the file content corresponding to the target application is in an updated state, then determining that the consumption status of the consumer group corresponding to the target application is active; otherwise, determining that the consumption status of the consumer group corresponding to the target application is inactive.
[0010] The second aspect of the present application provides an offset management device, including: an identification module for identifying a target method for submitting an offset by a target application; an acquisition module for obtaining the offset of a consumer group corresponding to the target application according to the target method, and analyzing the consumption status of the consumer group corresponding to the target application; and a storage module for storing the offset and consumption status in a preset database.
[0011] Optionally, the target mode includes a direct mode and an indirect mode, and the acquisition module is further used to: if the target mode is a direct mode, read the offset of the consumer group corresponding to the target application based on the target interface; if the target mode is an indirect mode, read the transit file list, and extract the consumer group corresponding to the target application and the offset corresponding to the consumer group in the transit file list.
[0012] Optionally, the acquisition module is further used to: determine a target directory path for storing the offset;
[0013] Read the staging file list based on the target directory path.
[0014] Optionally, the acquisition module is further used to: traverse the transit file list to read the file content of each file; parse the file content according to the file format of each file to obtain key fields, wherein the key fields include consumer group identifier, topic name, partition number and offset; match the consumer group related to the target application and the offset corresponding to the consumer group according to the identification information of the target application.
[0015] Optionally, the acquisition module is further used to: analyze the consumption status of the consumer group corresponding to the target application, including: checking the transit file list; if the file content corresponding to the target application is in an updated state, then the consumption status of the consumer group corresponding to the target application is determined to be active, otherwise the consumption status of the consumer group corresponding to the target application is determined to be inactive.
[0016] The third aspect of the present application provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the offset management method as described in the above embodiment.
[0017] The fourth aspect of the present application provides a computer-readable storage medium having a computer program or instruction stored thereon. When the computer program or instruction is executed, it is used to implement the offset management method as described in the above embodiment.
[0018] The fifth aspect of the present application provides a computer program product, including: a computer program or instructions, which, when executed, implements the offset management method as described in the above embodiment.
[0019] Therefore, this application has at least the following beneficial effects:
[0020] The embodiment of the present application can obtain the offset of the target application corresponding to the consumer group according to the target method of submitting the offset of the target application. For the direct method, the offset can be directly read using the interface; for the indirect method, the offset is obtained by reading and parsing the transit file from the external storage and storing it in the database, thereby realizing effective management and analysis of the offset data, so that downstream applications can better understand and track the overall consumption situation and achieve unified management. This solves the problem that the offset management method in the related art is only applicable to applications that actively submit offsets.
[0021] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS
[0022] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:
[0023] Figure 1 A flowchart of an offset management method provided according to an embodiment of the present application;
[0024] Figure 2 This is an example diagram of a method for managing an offset according to one embodiment of the present application;
[0025] Figure 3 This is an example diagram of obtaining a file system offset according to one embodiment of the present application;
[0026] Figure 4 is an example diagram of an offset management device according to an embodiment of the present application;
[0027] Figure 5Schematic diagram of the structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION
[0028] The following describes in detail embodiments of the present application, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.
[0029] The following describes the offset management method, device, electronic device and storage medium of the embodiment of the present application with reference to the accompanying drawings. In response to the problems mentioned in the above background technology, the present application provides an offset management method, in which the offset of the consumer group corresponding to the target application is obtained according to the target method of submitting the offset by the target application. For the direct method, the offset can be directly read using the interface; for the indirect method, the transfer file is read and parsed from the external storage to obtain the offset, and stored in the database, so as to achieve effective management and analysis of the offset data, so that downstream applications can better understand and track the overall consumption situation and achieve unified management. Thus, the problem that the offset management method in the related art is only applicable to applications that actively submit offsets is solved.
[0030] Specifically, Figure 1 A flowchart of an offset management method provided in an embodiment of the present application.
[0031] like Figure 1 As shown, the offset management method includes the following steps:
[0032] In step S101 , a target mode for a target application to submit an offset is identified.
[0033] Among them, the target method includes direct method and indirect method.
[0034] Understandably, Kafka consumers need to record offsets after consuming messages. Different applications may adopt different approaches to managing these offsets. Direct offset management involves submitting offsets directly back to Kafka, while indirect offset management involves writing offset information to an external storage system (such as a file system or database) rather than submitting it directly to Kafka.
[0035] In step S102, the offset of the consumer group corresponding to the target application is obtained according to the target mode, and the consumption status of the consumer group corresponding to the target application is analyzed.
[0036] It is understandable that if Figure 2As shown, for applications that submit offsets directly to Kafka, the existing Kafka open API is used to read the offsets of the application's consumer group and record the offsets of the consumer group. For applications that do not submit offsets directly to Kafka, the system integrates with the big data HDFS file system, scans the offset records submitted by the application in the specified directory, obtains the offset data, and records it. After aggregation, the offset data related to the application currently consuming the Kafka cluster is obtained and stored in the database. This aggregated offset data accurately reflects the current application's consumption status, consumption state, and consumption delay.
[0037] In one embodiment of the present application, the offset of the consumer group corresponding to the target application is obtained according to the target method, including: if the target method is a direct method, the offset of the consumer group corresponding to the target application is read based on the target interface; if the target method is an indirect method, the transfer file list is read, and the consumer group corresponding to the target application and the offset corresponding to the consumer group in the transfer file list are extracted.
[0038] Specifically, when the target application submits offsets directly, the offset data is stored internally in Kafka. Kafka provides an API to read the offsets of the application's consumer group. When the target application submits offsets indirectly, the offset data is stored in an external file system. These files, often called "transit files," store the consumer group's consumption offsets.
[0039] Furthermore, reading the transfer file list includes: determining a target directory path for storing the offset; and reading the transfer file list based on the target directory path.
[0040] In an embodiment of the present application, the transit files are typically stored in a specific directory. The directory path may be preset or specified through a configuration file or parameter. In an embodiment of the present application, the target directory path can be read from the configuration file. The file list under the target directory contains all transit files and may contain offset information for multiple consumer groups.
[0041] Furthermore, the consumer group corresponding to the target application in the transit file list and the offset corresponding to the consumer group are extracted, including: traversing the transit file list to read the file content of each file; parsing the file content according to the file format of each file to obtain key fields, wherein the key fields include the consumer group identifier, topic name, partition number and offset; matching the consumer group related to the target application and the offset corresponding to the consumer group according to the identification information of the target application.
[0042] It is understandable that the transfer file list usually contains multiple files, each of which may record the offset information of one or more consumer groups on a specific topic partition. These files may be generated by different applications, so their formats may vary, such as Figure 3 As shown, the embodiment of the present application can read a list of files and parse the file content according to the file format of each file in order to extract the required field information therefrom, including consumer group identifier, topic name, partition number and offset. In the parsed data, it is necessary to filter out the consumer group and its offset information related to the target application. The target application can be uniquely identified by its consumer group ID or other identification information. During the actual execution process, the embodiment of the present application can obtain the identification information (such as consumer group ID) of the target application from the configuration file, user input or other metadata, traverse the parsing results, and compare whether the consumer group identifier of each record matches the identification information of the target application. Thus, the consumer group and its offset information corresponding to a specific target application can be effectively extracted from the transit file list stored in the file system. It is not only applicable to a variety of file formats, but also provides a flexible matching mechanism to ensure that the required data can be accurately filtered out.
[0043] In one embodiment of the present application, the consumption status of the consumer group corresponding to the target application is analyzed, including: checking the transfer file list; if the file content corresponding to the target application is in an updated state, the consumption status of the consumer group corresponding to the target application is determined to be active, otherwise the consumption status of the consumer group corresponding to the target application is determined to be inactive.
[0044] It is understandable that the consumption status of a consumer group reflects whether it is working properly and continuously consuming messages. The active state indicates that the consumer group is processing new messages, while the inactive state may indicate that the consumer group has stopped working or has failed. The transit file records the consumption progress (offset) of the consumer group. If the file content is continuously updated, it means that the consumer group is still submitting the latest consumption progress; on the contrary, if the file has not been updated for a long time, it may mean that the consumer group has stopped consuming. Therefore, if Figure 3 As shown, the embodiment of the present application can effectively determine the consumption status of the target application corresponding to the consumer group by checking the transfer file list and analyzing the file update status. This method is simple and efficient, and is particularly suitable for application scenarios that rely on external file management systems to store offsets.
[0045] In step S103 , the offset and consumption status are stored in a preset database.
[0046] The preset database can be set according to actual conditions and is not specifically limited.
[0047] The embodiment of the present application can store the offset and consumption status in a preset database. Program developers or third-party management programs can directly obtain the data in the database and intuitively see the status of the current consumption program, consumption offset information, etc.
[0048] For example, the way the StructedStreaming framework obtains the offset is to first locate the offset submission directory (which is specified by the program and available to the user), such as Figure 3 As shown, first find an offsets file directory within the offsets directory. Open the offsets file directory, which contains a series of files. Locate the latest file. The last line of the file contains the partition and current offset information. This information is read and maintained in the result database, making it easily accessible to third-party applications.
[0049] It should be noted that the offset method of different frameworks is slightly different, and the file directory may also be different.
[0050] According to the offset management method proposed in the embodiment of the present application, the offset of the target application corresponding to the consumer group is obtained according to the target method of submitting the offset by the target application. For the direct method, the offset can be directly read using the interface; for the indirect method, the offset is obtained by reading and parsing the transit file from the external storage and storing it in the database, thereby realizing effective management and analysis of the offset data, so that downstream applications can better understand and track the overall consumption situation and achieve unified management. This solves the problem that the offset management method in the related art is only applicable to applications that actively submit offsets.
[0051] Next, the offset management device proposed in accordance with an embodiment of the present application will be described with reference to the accompanying drawings.
[0052] Figure 4 It is a block diagram of the offset management device of an embodiment of the present application.
[0053] like Figure 4 As shown, the offset management device 10 includes: an identification module 100 , an acquisition module 200 and a storage module 300 .
[0054] Among them, the identification module 100 is used to identify the target method for the target application to submit the offset; the acquisition module 200 is used to obtain the offset of the consumer group corresponding to the target application according to the target method, and analyze the consumption status of the consumer group corresponding to the target application; the storage module 300 is used to store the offset and consumption status to a preset database.
[0055] In one embodiment of the present application, the target mode includes a direct mode and an indirect mode, and the acquisition module 200 is further used to read the offset of the consumer group corresponding to the target application based on the target interface if the target mode is a direct mode; if the target mode is an indirect mode, read the transfer file list and extract the consumer group corresponding to the target application and the offset corresponding to the consumer group in the transfer file list.
[0056] In one embodiment of the present application, the acquisition module 200 is further used to: determine a target directory path for storing the offset; and read a transfer file list based on the target directory path.
[0057] In one embodiment of the present application, the acquisition module 200 is further used to traverse the transit file list and read the file content of each file; parse the file content according to the file format of each file to obtain key fields, wherein the key fields include a consumer group identifier, a topic name, a partition number, and an offset; match the consumer group related to the target application and the offset corresponding to the consumer group according to the identification information of the target application.
[0058] In one embodiment of the present application, the acquisition module 200 is further used to analyze the consumption status of the consumer group corresponding to the target application, including: checking the transfer file list; if the file content corresponding to the target application is in an updated state, then the consumption status of the consumer group corresponding to the target application is determined to be active, otherwise the consumption status of the consumer group corresponding to the target application is determined to be inactive.
[0059] It should be noted that the above explanation of the offset management method embodiment is also applicable to the offset management device of this embodiment, and will not be repeated here.
[0060] According to the offset management device proposed in the embodiment of the present application, the offset of the target application corresponding to the consumer group is obtained according to the target method of submitting the offset by the target application. For the direct method, the offset can be directly read using the interface; for the indirect method, the offset is obtained by reading and parsing the transit file from the external storage and storing it in the database, thereby realizing effective management and analysis of the offset data, so that downstream applications can better understand and track the overall consumption situation and achieve unified management. This solves the problem that the offset management method in the related art is only applicable to applications that actively submit offsets.
[0061] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device may include:
[0062] Memory 501 , processor 502 , and computer programs stored in the memory 501 and executable on the processor 502 .
[0063] When the processor 502 executes the program, the offset management method provided in the above embodiment is implemented.
[0064] Furthermore, the electronic device further includes:
[0065] The communication interface 503 is used for communication between the memory 501 and the processor 502 .
[0066] The memory 501 is used to store computer programs that can be run on the processor 502 .
[0067] The memory 501 may include a high-speed RAM (Random Access Memory) memory, and may also include a non-volatile memory, such as at least one disk memory.
[0068] If the memory 501, the processor 502, and the communication interface 503 are implemented independently, the communication interface 503, the memory 501, and the processor 502 can be connected to each other via a bus and communicate with each other. The bus can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5 Only one thick line is used in the diagram, but this does not mean that there is only one bus or one type of bus.
[0069] Optionally, in a specific implementation, if the memory 501, the processor 502 and the communication interface 503 are integrated on a chip, the memory 501, the processor 502 and the communication interface 503 can communicate with each other through an internal interface.
[0070] The processor 502 may be a CPU (Central Processing Unit), or an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of the present application.
[0071] An embodiment of the present application also provides a computer-readable storage medium having a computer program stored thereon, which implements the above offset management method when executed by a processor.
[0072] An embodiment of the present application also provides a computer program product, including: a computer program or instructions, which, when executed, implements the above offset management method.
[0073] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification and features of different embodiments or examples without contradiction.
[0074] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of technical features indicated. Thus, a feature specified as "first" or "second" may explicitly or implicitly include at least one such feature. In the description of this application, "N" means at least two, for example, two, three, etc., unless otherwise specifically defined.
[0075] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may be performed out of the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present application belong.
[0076] It should be understood that various parts of the present application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiment, the N steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application-specific integrated circuit having a suitable combination of logic gate circuits, a programmable gate array, a field programmable gate array, etc.
[0077] Those skilled in the art will understand that all or part of the steps in the method of the above embodiment can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiment.
[0078] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limitations on the present application. Ordinary technicians in this field can change, modify, replace and modify the above embodiments within the scope of the present application.
Claims
1. A method for managing an offset, characterized in that: The following steps are involved: Identify the target method for the target application to submit offsets; Obtaining the offset of the consumer group corresponding to the target application according to the target mode, and analyzing the consumption status of the consumer group corresponding to the target application; The offset and the consumption status are stored in a preset database.
2. The offset management method according to claim 1, characterized in that: The target mode includes a direct mode and an indirect mode, and obtaining the offset of the consumer group corresponding to the target application according to the target mode includes: If the target mode is the direct mode, reading the offset of the consumer group corresponding to the target application based on the target interface; If the target mode is the indirect mode, the transfer file list is read, and the consumer group corresponding to the target application and the offset corresponding to the consumer group in the transfer file list are extracted.
3. The offset management method according to claim 2, characterized in that: The reading of the transfer file list includes: Determine the target directory path for storing offsets; The transfer file list is read based on the target directory path.
4. The offset management method according to claim 2 or 3, characterized in that: Extracting the target application's corresponding consumer group and the offset corresponding to the consumer group in the transit file list includes: Traverse the transfer file list and read the file content of each file; Parsing the file content of each file according to the file format of each file to obtain key fields, wherein the key fields include a consumer group identifier, a topic name, a partition number, and an offset; The consumer group associated with the target application and the offset corresponding to the consumer group are matched according to the identification information of the target application.
5. The offset management method according to any one of claims 2 to 4, characterized in that: Analyze the consumption status of the consumer group corresponding to the target application, including: Checking the list of transfer documents; If the file content corresponding to the target application is in an updated state, the consumption state of the consumer group corresponding to the target application is determined to be an active state; otherwise, the consumption state of the consumer group corresponding to the target application is determined to be an inactive state.
6. An offset management device, characterized in that: include: An identification module, used to identify the target method for the target application to submit the offset; an acquisition module, configured to acquire an offset of a consumer group corresponding to the target application according to the target mode, and analyze a consumption status of the consumer group corresponding to the target application; A storage module is used to store the offset and the consumption status in a preset database.
7. The offset management device according to claim 6, characterized in that: The target mode includes a direct mode and an indirect mode, and the acquisition module is further used to: If the target mode is the direct mode, reading the offset of the consumer group corresponding to the target application based on the target interface; If the target mode is the indirect mode, the transfer file list is read, and the consumer group corresponding to the target application and the offset corresponding to the consumer group in the transfer file list are extracted.
8. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the offset management method according to any one of claims 1 to 5.
9. A computer-readable storage medium having a computer program or instruction stored thereon, characterized in that: When the computer program or instruction is executed, the offset management method according to any one of claims 1 to 5 is implemented.
10. A computer program product comprising: A computer program or instruction, characterized in that when the computer program or instruction is executed, it implements the offset management method described in any one of claims 1-5.