A method for indexing emails of a multi-user email box based on a KV system

Abstract

A method for indexing emails of a multi-user email box based on a KV system provided by the present invention includes: constructing a Server program architecture; constructing a Key-List mailbox structure based on the KV system according to the Server program architecture; constructing a Map structure thread lock bucket based on memory; and storing multiple field information of emails using a ProtoBuf structure. The addition of email information fields is flexible and efficient, effectively realizing backward / forward compatibility of the email information data field structure, and realizing the indexing of the summary information of user mailbox folders and email information.

Description

Technical Field

[0001] The present invention relates to the field of email servers, and particularly to a method for indexing emails of multiple-user email boxes based on a KV system. Background Art

[0002] One of the important functions of an email server is to store the mailbox information and email originals of multiple users. Users access the email server through MUA (such as webmail, pop, imap) clients to read the summary information and email original content of their personal mailboxes. With the sharp increase in the number of users and their emails, when the email server displays the summary information of personal mailboxes (folder information, number of emails) and reads the list of email originals, the speed becomes slower and slower, and the performance becomes lower and lower. Even in the case of high concurrency and high load, the email server may crash or be unable to provide services.

[0003] For the above reasons, it is necessary to cache or index the summary information of users' mailboxes and the list information of email originals, so as to quickly display the summary information of users' mailboxes and the list information of emails when users receive emails through MUA clients. At the same time, it is also necessary to improve the speed at which users traverse the email list, and when the number of emails in the user's mailbox changes (such as receiving new emails, deleting emails, moving emails from one folder to another, etc.), the summary information of the user's mailbox can be quickly updated and the email list information can be adjusted.

[0004] The mailbox formats of conventional email servers are mbox and maildir formats, which organize the storage of email originals in the form of a single file or a directory structure file respectively, without caching or indexing of relevant mailbox summary information. When it is necessary to display the mailbox summary information, the mbox format needs to summarize and count all the email original information and the number in a single file, and the maildir format needs to count the number of files and the file size in the entire directory. When changing or moving, it is also very complicated, so the efficiency is usually relatively low.

[0005] Most improved email servers still use the maildir format or cloud storage to store email originals, but at the same time, an indexing device is added to store the summary information of the mailbox and the list information of email originals to speed up the reading of relevant information.

[0006] One of the indexing devices stores the original email information of users in a binary structured index file. Each fixed-size structure block in the structured file records the relevant information of the original email (sender, recipient, subject, size, file ID, etc.). When reading information, by calculating the number of structure blocks in the structured file and traversing the structure block information, the relevant summary information is calculated. When displaying the email list information, the structure block information in the structured file can be directly read to achieve the effect of improving speed. Since the data structure of the structured block is fixed, if additional email information fields are needed, the entire structured file needs to be changed, which requires a very large amount of work whether in an online or offline manner.

[0007] Another indexing device stores the original email information of users in a MySQL database table. Using the email folder information table and email information table in the database, all users' email folder information (folder name, number of emails, etc.) and the relevant information of the original email (sender, recipient, subject, size, folder ID, file ID, etc.) are stored in the records of the database in the form of table fields. When reading information, queries and traversals are performed through relevant SQL statements. When data changes such as adding or deleting emails occur, due to the associated update relationship between the email folder information table and the email information table, in order to ensure atomic updates, database transaction operations need to be used, and the performance of database transaction operations is relatively low. At the same time, as the amount of user email data increases, there will also be a certain bottleneck in the query performance of the database. When adding email information fields, table locking operations are required. When the data volume is very large, long-term table locking is needed to update the new fields in the database.

[0008] In summary, the working efficiency of the indexing devices in the prior art is low. Summary of the Invention

[0009] In view of the above problems, the present invention is proposed to provide a multi-user email indexing method based on a KV system that overcomes the above problems or at least partially solves the above problems.

[0010] According to one aspect of the present invention, there is provided a multi-user email indexing method based on a KV system, including:

[0011] Construct a Server program architecture;

[0012] Construct a Key-List mailbox structure based on the KV system according to the Server program architecture;

[0013] Construct a Map structure thread lock bucket based on memory;

[0014] Use the ProtoBuf structure to store multiple field information of emails.

[0015] Optionally, the construction of the Server program architecture specifically includes:

[0016] The Server program is a multi-threaded service program, including a network IO thread pool for providing Restful interface access services of the HTTP protocol; a working thread pool for performing data query and update services of the KV storage system;

[0017] The network IO thread pool and the working thread pool interact through a message queue.

[0018] Optionally, the construction of the Key-List mailbox structure based on the KV system according to the Server program architecture specifically includes:

[0019] Threads in the working thread pool use multiple Key-Value data based on the KV storage system to construct the mailbox information and mail information list structure of Key-List. One piece of the Key-Value data is mailbox information data, and multiple pieces of the Key-Value data are mail information data. Each piece of the Key-Value data represents one piece of mail data and stores the relevant information of the mail;

[0020] Plan the content format of the Key in the Key-Value data. The Key includes a TAG identification field indicating the type of the Key, a mailbox ID and a mail ID field; the prefix part includes the same user ID;

[0021] The Value includes multiple fields. The Value of the mailbox information includes the mailbox update timestamp, the number of mails in the mailbox, the total Size of the mails in the mailbox, the largest mail ID and the smallest mail ID in the mailbox. The Value of the mail information includes the timestamp of the mail, the Size of the mail and the ProtoBuf serialized data of the summary information of the mail.

[0022] Optionally, the construction of the thread lock bucket of the Map structure based on memory specifically includes:

[0023] When threads in the working thread pool update the mailbox information and mail information of users, in order to ensure atomic operations of the data, user-level thread mutex locks are used;

[0024] A thread lock bucket of the Map structure based on memory is used;

[0025] When updating KV data, a batch operation interface of the KV system is adopted.

[0026] Optionally, the adoption of the ProtoBuf structure to store multiple field information of mails specifically includes:

[0027] The ProtoBuf structure stores multiple fields of email information. When storing into the Key-Value of the KV system, the entire structure is serialized into a string. When reading, this string is deserialized again, and the whole process is transparent to the user;

[0028] If new email information fields need to be added, only a field in the ProtoBuf structure needs to be added. During the deserialization process of old data, the newly added fields will automatically adopt default values. When updating, the newly added fields will be automatically serialized into the string. The whole process has no impact on the overall data, and the data conversion is flexible and efficient, realizing backward / forward compatibility of the data structure.

[0029] A multi-user email mailbox indexing method based on the KV system provided by the present invention includes: constructing a Server program architecture; constructing a Key-List mailbox structure based on the KV system according to the Server program architecture; constructing a Map structure thread lock bucket based on memory; and storing multiple field information of emails using the ProtoBuf structure. Adding new email information fields is flexible and efficient, effectively realizing backward / forward compatibility of the email information data field structure, and realizing the indexing of the summary information of the user mailbox folder and the email information.

[0030] The above description is only an overview of the technical solution of the present invention. In order to be able to understand the technical means of the present invention more clearly, it can be implemented according to the content of the specification. And in order to make the above and other purposes, features and advantages of the present invention more obvious and understandable, the following specifically gives the specific implementation manners of the present invention. Brief Description of the Drawings

[0031] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for the description of the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained according to these drawings without creative efforts.

[0032] Figure 1 It is a schematic diagram of the Server program architecture provided by the embodiment of the present invention;

[0033] Figure 2 It is a schematic diagram of the Key-List mailbox structure based on the KV system provided by the embodiment of the present invention;

[0034] Figure 3 It is a schematic diagram of the MAP structure thread lock bucket based on memory provided by the embodiment of the present invention. Detailed Description of the Invention

[0035] Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided so that the present disclosure can be more thoroughly understood and the scope of the present disclosure can be fully conveyed to those skilled in the art.

[0036] In the description of the embodiments of the present invention, the terms "comprising", "having" and any variations thereof in the specification, claims and drawings of the present invention are intended to cover non-exclusive inclusion. For example, a series of steps or units are included.

[0037] The technical solution of the present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments.

[0038] As Figure 1 shown, a Server program architecture is constructed: The Server program is a multi-threaded service program, mainly composed of two sets of thread pools. One set is a network IO thread pool, which provides Restful interface access services for the HTTP protocol. The other set is a worker thread pool, which performs data query and update services for the KV storage system. The network IO threads and the worker thread pool interact through a message queue.

[0039] An internal KV storage engine system is integrated, and index data such as the summary information of the mailboxes of multiple users and the mail information is stored in the KV system. The KV system is a very fast NoSQL storage system that can store a large amount of data and provide fast access to the data.

[0040] As Figure 2 shown, establishing a Key-List mailbox structure based on the KV system includes:

[0041] The working thread pool threads utilize multiple Key-Value data from the KV storage system to construct the mailbox information and the list structure of email information in the form of Key-List. Among them, one KV data is the mailbox information data, and the remaining multiple KV data are email information data. Each KV data represents an email data, storing various relevant information of the email. To establish the association relationship between the mailbox information and the email information, it is necessary to plan the content format of the Key in these KV data. The Key consists of multiple fields. The prefix part includes the same user ID to ensure that the data of the same user has the same prefix. At the same time, the Key also includes a TAG identification field indicating the type of the Key (mailbox information, email information), as well as the mailbox ID and email ID fields. The Value also consists of multiple fields. The Value of the mailbox information mainly consists of the mailbox update timestamp, the number of emails in the mailbox, the total size of the emails in the mailbox, the maximum email ID and the minimum email ID in the mailbox. The Value of the email information mainly consists of the email timestamp, the size of the email, and the Protobuf serialized data of the email summary information.

[0042] When adding or deleting an email, it is necessary to add or delete a piece of email Key-Value data and at the same time update the numerical values of the Count, Bytes, MaxID, and MinID fields in the Value of the mailbox Key-Value data.

[0043] When reading the mailbox information, only the Key-Value data of the mailbox needs to be read, which is a very fast reading operation in the NoSQL system, and the mailbox information can be directly obtained.

[0044] When traversing the emails in the mailbox, only need to sequentially read the Key-Value data in the NoSQL system according to the UID prefix and MailID order, and utilize the KV traversal pointer of the NoSQL system. Even in the case of massive data, the Key-Value data can be quickly read.

[0045] Construct a Key-List structure from multiple Key-Value data in the KV system, i.e., the Key is a record that stores the user's mailbox information, and the List is multiple records, each record storing the information of one user's email. In this way, the mailbox index structure of the user is constructed. When reading the mailbox information, only one Key-Value data needs to be read. Taking advantage of the NoSQL KV storage system, it can quickly locate and read a Key-Value mailbox information data among massive data, and the reading speed is very fast. When adding or deleting emails, only two records need to be updated simultaneously, one is the mailbox information, and the other is the added or deleted email information. And there is no need to use transactions. Instead, the user-level memory Mutex thread lock is used, and the batch operation (BatchPut) interface of the KV storage system is adopted to complete it at one time, with very high speed and efficiency.

[0046] As Figure 3 shown, constructing a thread lock bucket based on the in-memory Map structure includes: when the threads in the worker thread pool update the user's mailbox information and email information, in order to ensure the atomic operation of the data, the user-level (UID) thread mutex lock is used. At the same time, in order to meet the requirements of a large number of concurrent users updating data simultaneously, a thread lock bucket based on the in-memory Map structure is used. 10,000 buckets are pre-allocated. The element of the Map is the hash value of the UID, and the data corresponding to this element is the thread lock Mutex object. According to the characteristics of the Map data structure, the time complexity of a UID obtaining a lock is O(1) or O(logn). When updating the KV data, the batch operation interface (BatchPut) of the KV system is adopted to ensure the quick and atomic update of the data. At the same time, the automatic release function of the Mutex lock is implemented within the worker thread, effectively avoiding the situation of system deadlocks.

[0047] To ensure the atomicity of updating the Key-List data, a thread lock bucket based on the in-memory Map structure is used in the Server program, and the thread lock is obtained by hashing the Key and distributing it to different buckets. When the number of elements in the Map (i.e., the number of buckets) is set to 10,000 (or even more), the requirements of more user concurrent access can be met. The time complexity of the element positioning in the Map can reach O(1) or O(logn), which can quickly locate the specific lock and ensure the quick concurrent update of the data.

[0048] The adoption of the ProtoBuf structure to store multiple field information of emails includes: the ProtoBuf structure stores multiple fields of email information. When storing it into the Key-Value of the KV system, the entire structure is serialized into a string. When reading, this string is deserialized, and the whole process is transparent to the user. When new email information fields need to be added, only a field of the ProtoBuf structure needs to be added. During the deserialization process of the old data, the newly added fields will automatically adopt default values, which does not affect the integrity of the data. When updating, the newly added fields will be automatically serialized into the string, and the whole process has no impact on the overall data. The data conversion is flexible and efficient, effectively achieving backward / forward compatibility of the data structure.

[0049] For a Key-Value data of email information, multiple fields of the email information are serialized using the ProtoBuf structure and stored in the Value. During reading, deserialization is performed. When new fields are added, all the currently stored email information data does not need to be updated. The ProtoBuf structure will provide the default data values of the newly added fields. Only when the Key-Value data needs to be updated, the email information data stored in ProtoBuf will be updated, and it is updated on a single-piece basis, without affecting all the stored data. This dynamically flexible data storage method effectively improves the flexibility of storing and updating the data fields of email information.

[0050] Beneficial effects: Through multiple Key-Value data in the KV system, a mailbox index structure in the form of a Key-List for multiple users is constructed to realize the indexing of the folder summary information and email information of the user's mailbox, enabling fast query / update of mailbox information, as well as fast addition / deletion / update of email information.

[0051] Through the thread-lock bucket of the in-memory Map structure, the fast atomic update of mailbox information and email information in the Key-List structure of multiple users under high concurrency is solved.

[0052] Multiple fields of email information are stored in the Value of Key-Value in the form of ProtoBuf structure data. Adding new email information fields is flexible and efficient, effectively achieving backward / forward compatibility of the data field structure of email information.

[0053] The above specific implementation manners further elaborate on the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above are only specific implementation manners of the present invention and are not used to limit the protection scope of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for indexing emails in a multi-user email box based on a KV system, characterized in that, The indexing method includes: Construct a Server program architecture; Construct a Key-List mailbox structure based on the KV system according to the Server program architecture; Construct a Map structure thread lock bucket based on memory; Use the ProtoBuf structure to store multiple field information of emails.

2. The multi-user email message indexing method based on a KV system according to claim 1, wherein, The construction of the Server program architecture specifically includes: The Server program is a multi-threaded service program, including a network IO thread pool for providing Restful interface access services for the HTTP protocol; a working thread pool for data query and update services of the KV storage system; The network IO thread pool and the working thread pool interact through a message queue.

3. The multi-user email message indexing method based on a KV system according to claim 2, wherein, The construction of the Key-List mailbox structure based on the KV system according to the Server program architecture specifically includes: Threads in the working thread pool use multiple Key-Value data based on the KV storage system to construct the mailbox information and email information list structure of Key-List. One piece of the Key-Value data is mailbox information data, and multiple pieces of the Key-Value data are email information data. Each piece of the Key-Value data represents an email data and stores the relevant information of the email; Plan the content format of the Key in the Key-Value data. The Key includes a TAG identification field indicating the type of the Key, a mailbox ID field, and an email ID field; the prefix part includes the same user ID; The Value includes multiple fields. The Value of the mailbox information includes the mailbox update timestamp, the number of emails in the mailbox, the total Size of the emails in the mailbox, the largest email ID in the mailbox, and the smallest email ID. The Value of the email information includes the timestamp of the email, the Size of the email, and the ProtoBuf serialized data of the email summary information.

4. A multi-user email indexing method based on a KV system according to claim 1, characterized in that, The construction of the Map structure thread lock bucket based on memory specifically includes: When threads in the working thread pool update the mailbox information and email information of users, in order to ensure atomic operations of the data, user-level thread mutex locks are used; A Map structure thread lock bucket based on memory is used; When updating KV data, a batch operation interface of the KV system is adopted.

5. A method for indexing emails of a multi-user email box based on a KV system according to claim 1, characterized in that, The use of the ProtoBuf structure to store multiple field information of emails specifically includes: The ProtoBuf structure stores multiple fields of email information. When storing into the Key-Value of the KV system, the entire structure is serialized into a string. When reading, this string is deserialized again. The whole process is transparent to the user; If new email information fields need to be added, only a field of the ProtoBuf structure needs to be added. The newly added fields will automatically adopt default values during the deserialization process of the old data. When updating, the newly added fields will be automatically serialized into the string. The whole process has no impact on the overall data. The data conversion is flexible and efficient, realizing backward / forward compatibility of the data structure.

Images