Distributed container-based data mapping method, system, device and storage medium

Abstract

The application provides a distributed container-based data mapping method, system, device and storage medium, and the method comprises the following steps: converting original user accounts in each container into user accounts with the same format and length; rearranging the order of the user accounts, and replacing the actual storage value of the user accounts with the arranged order; when each server obtains new user account information, obtaining the cache index sequence of the next user account based on the global index and the current step length, and storing the cache index sequence in the cache of the server; taking the intersection of the user account set in the server which meets the label condition of a user type and the local cache index sequence of the server; and establishing a bit-based mapping relationship based on the intersection and the label of the user type. The application can simplify a large population data set, greatly improve the query efficiency and update efficiency, and reduce the stability impact of massive data random update on the system.

Description

Technical Field

[0001] This invention relates to the field of data storage management, and more specifically, to a data mapping method, system, device, and storage medium based on distributed containers. Background Technology

[0002] With the rapid development of internet technology, more and more data is being stored on servers and frequently needs to be queried. Currently, one method for data storage is to segment datasets based on user demographics. For example, if a cargo owner selects on an internet platform that they need refrigerated trucks to transport agricultural products from Shanghai, the server needs to query driver data from the dataset containing users in Shanghai and those using refrigerated trucks.

[0003] Because servers store massive amounts of data, network platforms face the following technical challenges:

[0004] 1. The distribution of users within a single user group is extremely uneven, ranging from tens or hundreds of users in some groups to millions or even tens of millions. In query services, a single dataset stores a single user group, and due to the vast differences in data volume, query efficiency is highly unstable, facing the risk of being overwhelmed by high QPS (queries per second) queries.

[0005] 2. Storage devices consume a huge amount of space on a dataset of millions of people. Calculating the full data of tens of thousands of people will consume more than 10TB of storage. Moreover, this data is refreshed every day, resulting in extremely high I / O, which will pose a great challenge to the stability of online production.

[0006] 3. With hundreds of new, modified, and deleted users every day, the stability of the system is severely tested.

[0007] Based on the above considerations, it is necessary to streamline the population data, improve query efficiency and update efficiency, and reduce the stability impact caused by massive data updates.

[0008] Therefore, the present invention provides a data mapping method, system, device and storage medium based on distributed containers. Summary of the Invention

[0009] To address the problems in the prior art, the present invention aims to provide a data mapping method, system, device, and storage medium based on distributed containers, which overcomes the difficulties of the prior art, can streamline ultra-large population datasets, greatly improve query efficiency and update efficiency, and reduce the impact of random updates of massive data on system stability.

[0010] Embodiments of the present invention provide a data mapping method based on a distributed container, comprising the following steps:

[0011] Convert the original user accounts in each container into user accounts with the same format and length;

[0012] Reorder the user accounts and replace the actual stored values ​​of the user accounts with the rearranged order;

[0013] When each server obtains information about a newly added user account, it obtains the cache index sequence of the next user account based on the global index and the current step size, and stores it in the cache of the server.

[0014] Take the intersection of the set of user accounts in the server that satisfy the tag conditions of a user type with the local cache index sequence of the server; and

[0015] A bit-based mapping relationship is established based on the intersection and the user type labels.

[0016] Preferably, converting the original user accounts in each container into user accounts of the same format and length includes:

[0017] Convert the original user accounts in each container to hexadecimal to obtain user accounts with the same format and length.

[0018] Preferably, the rearrangement of user accounts and the replacement of the actual stored values ​​of the user accounts with the rearranged order further includes:

[0019] Before starting the server, an auto-incrementing counter is used to sequentially set a startup sequence number for each server in the server cluster, which serves as the current step size for orchestrating the user account index.

[0020] Preferably, the server cluster comprises ≥20 servers.

[0021] Preferably, the step of obtaining the cached index sequence of the next user account based on the global index and the current step size when each server obtains new user account information, and storing it in the cache of the server, further includes:

[0022] Each server obtains a new user account index registration using a cluster deployment method, which divides the cluster into shards using self-incrementing technology units, and adds the user account to the cluster shard.

[0023] Preferably, the step of taking the intersection of the set of user accounts in the server that satisfy the tag conditions of a user type and the local cache index sequence of the server further includes...

[0024] Based on the user type tag conditions uploaded by the server, a set of user accounts that meet the tag conditions is selected from the container cluster.

[0025] Take the intersection of the set of user accounts and the local cache index sequence of the server.

[0026] Preferably, the step of establishing a bit-based mapping relationship based on the intersection and the user type tags includes...

[0027] The system sequentially checks whether each account in the current user account set appears in the local cache index sequence of the server. If it does, the bit that is mapped to the account is marked as 1; if it does not appear, the bit that is mapped to the account is marked as 0.

[0028] Preferably, it further includes: when the server receives the tag conditions of the user type, it performs a retrieval based on the index in the intersection based on the mapping relationship.

[0029] Embodiments of the present invention also provide a data mapping system based on distributed containers, used to implement the above-described data mapping method based on distributed containers, wherein the data mapping system based on distributed containers includes:

[0030] The format conversion module converts the original user accounts in each container into user accounts with the same format and length.

[0031] The account sorting module rearranges the order of user accounts and replaces the actual stored value of the user accounts with the rearranged order;

[0032] The cache index sequence module, when each of the servers obtains new user account information, obtains the cache index sequence of the next user account based on the global index and the current step size, and stores it in the cache of the server;

[0033] The index intersection module takes the intersection of the set of user accounts in the server that satisfy the tag conditions of a user type with the local cache index sequence of the server; and

[0034] The mapping module establishes a bit-based mapping relationship based on the intersection and the user type's label.

[0035] Embodiments of the present invention also provide a data mapping device based on a distributed container, comprising:

[0036] processor;

[0037] A memory in which executable instructions of the processor are stored;

[0038] The processor is configured to execute the steps of the above-described data mapping method based on distributed containers by executing the executable instructions.

[0039] Embodiments of the present invention also provide a computer-readable storage medium for storing a program that, when executed, implements the steps of the above-described data mapping method based on a distributed container.

[0040] The purpose of this invention is to provide a data mapping method, system, device, and storage medium based on distributed containers, which can streamline ultra-large population datasets, greatly improve query efficiency and update efficiency, and reduce the impact of random updates of massive data on system stability. Attached Figure Description

[0041] Other features, objects, and advantages of the invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings.

[0042] Figure 1 This is a flowchart of the data mapping method based on distributed containers of the present invention.

[0043] Figures 2 to 5 This is a schematic diagram illustrating the implementation process of the data mapping method based on distributed containers of the present invention.

[0044] Figure 6 This is a schematic diagram of the data mapping system based on distributed containers of the present invention.

[0045] Figure 7 This is a schematic diagram of the structure of the data mapping device based on distributed containers of the present invention.

[0046] Figure 8 This is a schematic diagram of the structure of a computer-readable storage medium according to an embodiment of the present invention. Detailed Implementation

[0047] The following specific examples illustrate the implementation methods of this application. Those skilled in the art can easily understand the other advantages and effects of this application from the content disclosed herein. This application can also be implemented or applied through other different specific embodiments, and various details in this application can be modified or changed according to different viewpoints and application systems without departing from the spirit of this application. It should be noted that, unless otherwise specified, the embodiments and features in the embodiments of this application can be combined with each other.

[0048] The embodiments of this application will now be described in detail with reference to the accompanying drawings, so that those skilled in the art can easily implement the application. This application may be embodied in many different forms and is not limited to the embodiments described herein.

[0049] In this application, the terms "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics represented in connection with that embodiment or example, which are included in at least one embodiment or example of this application. Furthermore, the specific features, structures, materials, or characteristics represented may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate different embodiments or examples represented in this application, as well as features of different embodiments or examples.

[0050] Furthermore, the terms "first" and "second" are used for illustrative purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the representation of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0051] For the purpose of clearly describing this application, devices that are not relevant to the description are omitted, and the same or similar components throughout the specification are given the same reference numerals.

[0052] Throughout this specification, when it is said that a device is "connected" to another device, this includes not only "direct connection" but also "indirect connection" by placing other components in between. Furthermore, when it is said that a device "comprises" a certain constituent element, unless otherwise stated otherwise, this does not exclude other constituent elements, but rather implies that other constituent elements may be included.

[0053] When we say that a device is "above" another device, this can mean that it is directly above the other device, or it can mean that other devices are present in between. Conversely, when we say that a device is "directly" "above" another device, there are no other devices present in between.

[0054] Although the terms first, second, etc., are used in some instances herein to refer to various elements, these elements should not be limited by these terms. These terms are used only to distinguish one element from another. For example, first interface and second interface, etc., are used. Furthermore, as used herein, the singular forms “a,” “an,” and “the” are intended to also include the plural forms unless the context indicates otherwise. It should be further understood that the terms “comprising,” “including,” indicate the presence of features, steps, operations, elements, components, items, kinds, and / or groups, but do not exclude the presence, occurrence, or addition of one or more other features, steps, operations, elements, components, items, kinds, and / or groups. The terms “or” and “and / or” as used herein are interpreted as inclusive, or mean any one or any combination thereof. Thus, “A, B, or C” or “A, B, and / or C” means “any one of: A; B; C; A and B; A and C; B and C; A, B, and C.” Exceptions to this definition will only occur if the combination of elements, functions, steps, or operations is inherently mutually exclusive in some way.

[0055] The technical terms used herein are for reference only to specific embodiments and are not intended to limit the scope of this application. The singular form used herein includes the plural form unless the statement explicitly indicates otherwise. The word "comprising" as used in the specification means to specify a particular characteristic, region, integer, step, operation, element, and / or component, and does not exclude the presence or addition of other characteristics, regions, integers, steps, operations, elements, and / or components.

[0056] Although not explicitly defined, all terms, including technical and scientific terms used herein, shall have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. Terms defined in commonly used dictionaries shall be further interpreted as having a meaning consistent with the relevant technical literature and the content of this present application, and shall not be over-interpreted as having an ideal or overly formulaic meaning unless otherwise defined.

[0057] Figure 1 This is a flowchart of the data mapping method based on distributed containers of the present invention. Figure 1 As shown, an embodiment of the present invention provides a data mapping method based on a distributed container, comprising the following steps:

[0058] S110. Convert the original user accounts in each container into user accounts of the same format and length.

[0059] S120. Rearrange the order of user accounts and replace the actual stored value of the user accounts with the rearranged order;

[0060] S130. When each server obtains information about a newly added user account, it obtains the cache index sequence of the next user account based on the global index and the current step size, and stores it in the server's cache.

[0061] S140. Take the intersection of the set of user accounts in the server that meet the tag conditions of a user type with the local cache index sequence of the server; and

[0062] S150. Establish bit-based mapping relationships based on the intersection and user type tags.

[0063] This invention employs distributed storage technology, utilizing the disk space on every machine within an enterprise via a network and combining these dispersed storage resources into a virtual storage device. Data is distributed and stored across various parts of the enterprise. This is achieved by distributing data across multiple independent devices. Traditional network storage systems use a centralized storage server to store all data, making the storage server a bottleneck for system performance and a focal point for reliability and security, failing to meet the needs of large-scale storage applications. The distributed network storage system adopts a scalable system architecture, utilizing multiple storage servers to share the storage load and using location servers to locate stored information. This not only improves system reliability, availability, and access efficiency but also facilitates expansion.

[0064] In a preferred embodiment, step S110 includes: converting the original user accounts in each container into hexadecimal to obtain user accounts with the same format and length.

[0065] In a preferred embodiment, step S120 includes: before starting the server, setting a startup sequence number for each server in the server cluster sequentially using an incrementing counter, as the current step size for orchestrating the user account index.

[0066] In a preferred embodiment, the server cluster comprises ≥20 servers.

[0067] In a preferred embodiment, step S130 further includes:

[0068] Each server obtains a new user account index for registration using a cluster deployment method. The cluster mode is divided into shards using self-incrementing technology units, and new user accounts are added in the cluster mode shards.

[0069] In a preferred embodiment, step S140 further includes:

[0070] S141. Based on the tag conditions of the user type uploaded by the server, filter out a set of user accounts that meet the tag conditions in the container cluster.

[0071] S142. Take the intersection of the user account set and the local cache index sequence of the server.

[0072] In a preferred embodiment, step S150 includes: sequentially determining whether each account in the current user account set appears in the local cache index sequence of the server; if so, marking the bit that has a mapping relationship with the account as 1; if not, marking the bit that has a mapping relationship with the account as 0.

[0073] In a preferred embodiment, it further includes:

[0074] Step S160: When the server receives the tag conditions of the user type, it performs a retrieval based on the index in the intersection of the mapping relationship.

[0075] refer to Figure 2 As shown, this invention mainly provides a method for data mapping compression, converting Set into Map and transforming a single structure into a composite structure; when users are repeated, only one user mapping relationship needs to be maintained, so that the same mapping relationship can be reused in multiple groups of people to the greatest extent.

[0076] Extreme compression of user index: Considering the special nature of bitmap index, the key can be directly transferred to the offset. At the same time, for the actual result, only one bit, i.e., 0 or 1, is needed to mark the current user's status. For example, 0 means not in the crowd, and 1 means in the crowd.

[0077] Advantages of global indexes: Global indexes allow you to specify your own partitioning method for the indexed data, and the partition key of the index is always a subset of the index keys. For a given index key, since it includes all index partition keys, its data can only fall within a fixed index partition. Therefore, as long as uniqueness constraints are guaranteed within each index partition, uniqueness constraints can be guaranteed across the entire table. Furthermore, since there is only one index data structure within a single index partition, it is easy to guarantee uniqueness constraints, thus solving the problem.

[0078] Mapping Compression: To ensure compatibility with bitmap indexes, the user mapping relationship is constructed in a way that aligns with the offset. At the outset, user UIDs are converted to hexadecimal to guarantee that user UIDs across different services have the same format and length. Simultaneously, a UID generator is used to rearrange the user ID order from one to ten million, replacing the actual stored value with this rearrangement order, thus maximizing compression.

[0079] Tags and user groups have a corresponding relationship. For example, tags such as "refrigerated truck," "located in Shanghai," and "B1 driver's license" correspond to a group of drivers. Both tags and user groups are essentially based on underlying user tagging data. The difference lies in their usage: tags are primarily used for display and querying, requiring high query scalability and data structuring, but less so for QPS and real-time performance. Therefore, tagging capabilities focus primarily on flexibility. User groups, on the other hand, are used for real-time scenario processing, with a simpler data structure but extremely high QPS and real-time performance requirements. Therefore, ensuring high performance is the core metric for this approach.

[0080] When selecting a technical solution, the unique characteristics of big data necessitate careful consideration of the data production and storage processes. Internet platforms often handle hundreds of millions of data points, making persistence and caching crucial for supporting various service scenarios. How to store tens of millions of data points and ensure timely data updates significantly impacts the overall solution design.

[0081] The following will focus on the computing, storage, and service-oriented architecture, especially the key parts, and will provide a detailed explanation of how to improve the query efficiency of tags and user groups.

[0082] When dealing with new user additions in a distributed environment, maintaining a global index is a crucial issue that must be addressed. This can be addressed using the distributed snowflake ID generation algorithm (see...). Figure 3 As seen in the previous section, using the machine's own ID to generate a unique ID is a good approach. Combining this approach with the business model of this scenario, we can use Redis counters to arrange the startup sequence number of each machine from 1 to n at the beginning of each service startup, and use this as the step number arranged by the user.

[0083] refer to Figure 4 As shown, the distributed index orchestration algorithm is as follows: When each service obtains new user information, it calculates the orchestration index of the next user based on the global index and step, i.e., {index+1, index+2...index+n}, and caches the calculation result locally.

[0084] refer to Figure 5As shown, during the population calculation on the cluster, a set of user accounts that meet the tag conditions of user type are selected based on the selected conditions. This set is then intersected with the index sequence (3), and the actual bitmap is constructed. Bitmap indexing refers to bitmap index technology, a special type of database indexing technology. Its index uses bit arrays (or bitmap, bit set, bit string, bit vector) for storage and calculation operations. Bitmap indexing is a special type of database index that uses bitmaps. It is mainly created for columns with a large number of identical values ​​(e.g., category, operator, department ID, warehouse ID, etc.). An index row in an index block stores the key value and the start and end Rowid, as well as the position code of these key values. Each bit in the position code indicates the presence or absence of the data row corresponding to the key value. A block may point to the position of dozens or even hundreds or thousands of rows of data.

[0085] Bitmap indexes are highly efficient for selecting multiple keys. Consider the following query: Assume `col1` and `col2` are attributes with low selectivity (i.e., only a limited number of enumerated values, like gender). We can create bitmap indexes for each. To find records that satisfy the SQL query, we perform a bitwise AND operation between the bitmap of `col1`'s `xx` value and the bitmap of `col2`'s `yy` value. The intersection bitmap indicates which record satisfies the query condition. For example, if the database has five records, with `xx`'s bitmap being 01101 and `yy`'s bitmap being 01000, the second record in the intersection bitmap is the query record. Another use case for bitmap indexes is counting the tuple of given selection conditions, such as the number of records satisfying the query condition. This can be directly calculated using the bitmap bitwise operations, avoiding the overhead of actually retrieving the data. Deleting records creates gaps between sequentially arranged records, as moving records to fill these gaps is extremely costly. To identify deleted records, an existence bitmap can be stored. In this bitmap, if the value of the i-th bit is 0, it indicates that record i does not exist; otherwise, it is 1. The insertion of a record should not affect the sequential numbering of other records. Therefore, the insertion operation can be accomplished by appending a record to the end of the file or replacing the deleted record.

[0086] The biggest advantage of bitmap indexes is their extremely small space footprint. Taking the `gender` attribute as an example, indexing 10 million records requires only 10 million bits, or approximately 1.2MB of storage space. Because this attribute has two enumerated values, `m` and `f`, the total storage space needed is only 2.4MB, which can be directly loaded into memory. The bitwise operations required for data filtering are also highly efficient. Assuming a table has 1 million records, each bitmap will contain 1 million bits, equivalent to 128KB. Assuming a word length of 32 bits, calculating the intersection of two bitmaps requires only 31,250 instructions. Since modern computers can perform billions of instructions per second, bitwise operations are extremely fast. For operations involving counting records using bitmaps, a lookup table method can be used to speed up the counting process. The lookup table method involves setting up a mapping table that stores the mapping relationship between the table's numerical index and the number of 1s in its corresponding binary representation. For example, maintaining an array with 256 items, each with 8 bits, where the i-th item stores the number of 1s in the binary representation of i. Bitmap has the following technical characteristics:

[0087] 1. Storage space of Bitmap index

[0088] Compared to B*Tree indexes, bitmap indexes occupy very little space because they only store the start and end Rowid of the key value and the bitmap.

[0089] The space usage of a bitmap is mainly related to the following four factors:

[0090] a. Total number of records in the table

[0091] b. The number of key values ​​in the indexed column; the fewer distinct values ​​in the column, the fewer bitmaps are required.

[0092] c. Operation type: Batch insertion requires far fewer bitmaps than single insertion. This was the case in 8i and 9i, but not in 10G. See the analysis below for details.

[0093] d. Physical distribution of the same key value in the index column: In versions 8i and 9i, data with the same key value in different blocks will be represented by different bitmap rows (segments).

[0094] 2. Bitmap index creation speed

[0095] Bitmap indexes do not require sorting during creation and are stored bit-by-bit, requiring less space.

[0096] B*Tree indexes, on the other hand, require sorting and positioning operations during creation, making them much slower.

[0097] 3. Bitmap indexes allow null keys.

[0098] Because B*Tree indexes do not record null values, a full table scan will be performed when using a query based on IS NULL.

[0099] When performing an IS NULL query on a bitmap indexed column, the index can be used.

[0100] 4. Efficient access to table records using Bitmap indexes

[0101] When using count(XX), statistical data can be quickly obtained by directly accessing the index.

[0102] When performing AND, OR, or IN(x,y,...) queries based on columns of a bitmap index, the bitmap index is used directly for the OR operation, allowing data to be filtered before accessing the data.

[0103] 5. Bitmap indexing only requires one indexing for batch DML operations.

[0104] Because the data is reflected through bitmaps, updating the index during batch operations is much faster than processing the B*Tree index row by row.

[0105] 6. Locking mechanism for Bitmap indexes

[0106] For B*Tree indexes, insert operations will not lock DML operations in other sessions.

[0107] Bitmap indexes, on the other hand, use bitmaps to reflect data. When different sessions update the same bitmap segment with the same key value, operations such as insert, update, and delete will cause locking.

[0108] For Oracle 8i and 9i, when inserting a single row, since a bitmap row (bitmap segment) only records 8 rows of data, a maximum of 8 rows of data with the same key value can be locked during DML operations.

[0109] During batch insertion, just like in 10G, there is only one bitmap row (bitmap segment) for the same key value. Therefore, DML operations on all data with the same key value will be locked.

[0110] Furthermore, in terms of service deployment, the registration of new user indexes adopts a cluster deployment method, with Redis used for cluster sharding. For the query service, which is designed for network-intensive queries, a sufficient number of machines are required to provide the service. Therefore, the entire query service needs to be deployed in a cluster of more than 20 containers.

[0111] The volume of matching requests for cargo is very high, reaching thousands of requests per second during the morning peak. A single cargo shipment requires the acquisition of driver tags for hundreds or even thousands of driver data points. Therefore, the volume of driver feature data requests is an enormous challenge. It is conservatively estimated that the peak number of requests per second needs to reach at least one million. Simply increasing the number of servers is neither economical nor scalable, and is a poor solution. The best approach is to reasonably and accurately identify technical bottlenecks and performance shortcomings, and overcome them through clever design.

[0112] After the implementation of this invention, the storage space for user index data in Redis is very stable for online data services based on user tags, generally maintained at 2-8MB. For all 20,000+ users, the total storage usage is less than 8GB, representing a nearly 100-fold improvement in efficiency compared to the storage usage before compression.

[0113] Even more valuable is the fact that, in terms of query efficiency and stability, the query efficiency after implementing bitmap index compression is far higher than the original set; and regardless of the size of the result set, the query time is very stable, ensuring the stability of the entire service during high QPS (100,000 times / second).

[0114] The purpose of this invention is to provide a data mapping method based on distributed containers, which can simplify ultra-large population datasets, greatly improve query efficiency and update efficiency, and reduce the impact of random updates of massive data on system stability.

[0115] Figure 6 This is a schematic diagram of the data mapping system based on distributed containers according to the present invention. Figure 6 As shown, embodiments of the present invention also provide a data mapping system based on a distributed container, used to implement the above-described data mapping method based on a distributed container. The data mapping system based on a distributed container includes:

[0116] The format conversion module 51 converts the original user accounts in each container into user accounts with the same format and length.

[0117] Account sorting module 52 rearranges the order of user accounts and replaces the actual stored value of the user accounts with the rearranged order;

[0118] The cache index sequence module 53, when each server obtains new user account information, obtains the cache index sequence of the next user account based on the global index and the current step size, and stores it in the server's cache;

[0119] Index intersection module 54 intersects the set of user accounts in the server that meet the tag conditions of a user type with the local cache index sequence of the server; and

[0120] The mapping module 55 establishes bit-based mapping relationships based on the intersection and user type labels.

[0121] In a preferred embodiment, the format conversion module 51 is configured to convert the original user accounts in each container into hexadecimal format to obtain user accounts with the same format and length.

[0122] In a preferred embodiment, the account sorting module 52 is configured to sequentially set a startup sequence number for each server in the server cluster using an incrementing counter before starting the server, as the current step size for sorting the user account index.

[0123] In a preferred embodiment, the server cluster comprises ≥20 servers.

[0124] In a preferred embodiment, the cache index sequence module 53 is configured to use a cluster deployment method for each server to obtain the newly added user account index registration, and to divide the cluster mode into shards through self-incrementing technology units, and to add user accounts in the cluster mode shards.

[0125] In a preferred embodiment, the index intersection module 54 is configured to filter out a set of user accounts that meet the tag conditions in the container cluster based on the tag conditions of the user types uploaded by the server; and to take the intersection of the set of user accounts with the local cache index sequence of the server.

[0126] In a preferred embodiment, the mapping module 55 is configured to sequentially determine whether each account in the user account set appears in the local cache index sequence of the server. If it does, the bit with the mapping relationship with the account is marked as 1; if it does not appear, the bit with the mapping relationship with the account is marked as 0.

[0127] The distributed container-based data mapping system of the present invention can simplify ultra-large population datasets, greatly improve query efficiency and update efficiency, and reduce the impact of random updates of massive data on system stability.

[0128] This invention also provides a data mapping device based on a distributed container, including a processor and a memory storing executable instructions for the processor. The processor is configured to execute steps of a distributed container-based data mapping method by executing the executable instructions.

[0129] As shown above, the distributed container-based data mapping system of this invention in this embodiment can streamline ultra-large population datasets, greatly improve query efficiency and update efficiency, and reduce the impact of random updates of massive data on the stability of the system.

[0130] Those skilled in the art will understand that various aspects of the present invention can be implemented as systems, methods, or program products. Therefore, various aspects of the present invention can be specifically implemented in the following forms: a completely hardware implementation, a completely software implementation (including firmware, microcode, etc.), or a combination of hardware and software aspects, collectively referred to herein as a "circuit," "module," or "platform."

[0131] Figure 7 This is a schematic diagram of the structure of the data mapping device based on distributed containers of the present invention. See below for reference. Figure 7 To describe an electronic device 600 according to this embodiment of the present invention. Figure 7 The electronic device 600 shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.

[0132] like Figure 7 As shown, the electronic device 600 is presented in the form of a general-purpose computing device. The components of the electronic device 600 may include, but are not limited to: at least one processing unit 610, at least one storage unit 620, a bus 630 connecting different platform components (including storage unit 620 and processing unit 610), a display unit 640, etc.

[0133] The storage unit stores program code, which can be executed by the processing unit 610 to perform the steps described in the above-described section on the electronic prescription transfer processing method according to various exemplary embodiments of the present invention. For example, the processing unit 610 can perform actions such as... Figure 1 The steps are shown in the figure.

[0134] Storage unit 620 may include a readable medium in the form of a volatile storage unit, such as random access memory (RAM) 6201 and / or cache memory 6202, and may further include a read-only memory (ROM) 6203.

[0135] Storage unit 620 may also include a program / utility 6204 having a set (at least one) program module 6205, such program module 6205 including but not limited to: operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0136] Bus 630 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the multiple bus structures.

[0137] Electronic device 600 can also communicate with one or more external devices 700 (e.g., keyboard, pointing device, Bluetooth device, etc.), and with one or more devices that enable a user to interact with electronic device 600, and / or with any device that enables electronic device 600 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 650. Furthermore, electronic device 600 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 660. Network adapter 660 can communicate with other modules of electronic device 600 via bus 630. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage platforms.

[0138] This invention also provides a computer-readable storage medium for storing a program, which, when executed, implements the steps of a distributed container-based data mapping method. In some possible implementations, various aspects of the invention can also be implemented as a program product comprising program code that, when run on a terminal device, causes the terminal device to perform the steps described in the above-described electronic prescription processing method section of this specification according to various exemplary embodiments of the invention.

[0139] As shown above, the distributed container-based data mapping system of this invention in this embodiment can streamline ultra-large population datasets, greatly improve query efficiency and update efficiency, and reduce the impact of random updates of massive data on the stability of the system.

[0140] Figure 8 This is a schematic diagram of the structure of the computer-readable storage medium of the present invention. (Reference) Figure 8 As shown, a program product 800 for implementing the above-described method according to an embodiment of the present invention is described. This product may employ a portable compact disc read-only memory (CD-ROM) and include program code, and may run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto. In this document, the readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0141] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0142] Computer-readable storage media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable storage medium may also be any readable medium other than a readable storage medium that can transmit, propagate, or transfer a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0143] Program code for performing the operations of this invention can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0144] In summary, the purpose of this invention is to provide a data mapping method, system, device, and storage medium based on distributed containers, which can streamline ultra-large population datasets, greatly improve query efficiency and update efficiency, and reduce the impact of random updates of massive data on system stability.

[0145] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various simple deductions or substitutions can be made without departing from the concept of the present invention, and all such modifications and substitutions should be considered within the scope of protection of the present invention.

Claims

1. A data mapping method based on distributed containers, characterized in that, Includes the following steps: Convert the original user accounts in each container to hexadecimal to obtain user accounts with the same format and length. The order of user accounts is rearranged and replaced with the actual stored value of the user accounts. Before starting the server, an auto-incrementing counter is used to set the startup sequence number for each server in the server cluster, which serves as the current step size for rearranging the user account index. When each server obtains new user account information, it obtains the cache index sequence of the next user account based on the global index and the current step size, and stores it in the cache of the server. Each server obtains the new user account index registration in a cluster deployment mode, divides the cluster mode into shards through self-incrementing technology units, and adds the user account in the cluster mode shard. Take the intersection of the set of user accounts in the server that meet the tag conditions of a user type with the local cache index sequence of the server; as well as The system sequentially checks whether each account in the current user account set appears in the local cache index sequence of the server. If it does, the bit that is mapped to the account is marked as 1; if it does not appear, the bit that is mapped to the account is marked as 0.

2. The data mapping method based on distributed containers as described in claim 1, characterized in that, The server cluster consists of ≥20 servers.

3. The data mapping method based on distributed containers as described in claim 1, characterized in that, The step of taking the intersection of the set of user accounts in the server that meet the tag conditions of a user type with the local cache index sequence of the server also includes... Based on the user type tag conditions uploaded by the server, a set of user accounts that meet the tag conditions is selected from the container cluster. Take the intersection of the set of user accounts and the local cache index sequence of the server.

4. A data mapping system based on distributed containers, used to implement the data mapping method based on distributed containers as described in claim 1, characterized in that, include: The format conversion module converts the original user accounts in each container into hexadecimal format to obtain user accounts with the same format and length. The account sorting module rearranges the order of user accounts and replaces the actual stored value of the user accounts with the rearranged order. Before starting the server, it sets the start sequence number for each server in the server cluster through an auto-incrementing counter, which serves as the current step size for rearranging the user account index. The cache index sequence module obtains the cache index sequence of the next user account based on the global index and the current step size when each server obtains new user account information, and stores it in the cache of the server. The registration of new user account index by each server adopts a cluster deployment method, divides the cluster mode into shards through the self-incrementing technology unit, and adds the user account in the cluster mode shard. The index intersection module takes the intersection of the set of user accounts in the server that meet the tag conditions of a user type with the local cache index sequence of the server; as well as The mapping module sequentially determines whether each account in the user account set appears in the local cache index sequence of the server. If it does, the bit with the mapping relationship with the account is marked as 1; if it does not appear, the bit with the mapping relationship with the account is marked as 0.

5. A data mapping device based on distributed containers, characterized in that, include: processor; A memory in which executable instructions of the processor are stored; The processor is configured to execute the steps of the data mapping method based on any one of claims 1 to 3 via executing the executable instructions.

6. A computer-readable storage medium for storing a program, characterized in that, When the program is executed by the processor, it implements the steps of the data mapping method based on distributed containers as described in any one of claims 1 to 3.

Images