Data sharding read-write method, system, device and storage medium in key-value storage

Abstract

The application provides a data sharding read-write method, system, device and storage medium in a key-value storage, wherein the method comprises the following steps: periodically scanning software installed in a commercial terminal based on a software scanning management module and feeding back software copyright management information to a server; judging, by the server, whether a software version corresponding to the software copyright management information is a paid version or a free version based on a preset software version directory; and performing an uninstallation task according to the judgment result. The application can automatically count enterprise commercial software installation details, monitor the installation of risky software, mark the details of paid software that has not been purchased by the enterprise, provide data support for cleaning work, and automatically update the paid version to the latest free version.

Description

Technical Field

[0001] This invention relates to the field of data storage, and more specifically, to a method, system, device, and storage medium for data fragmentation reading and writing in key-value stores. Background Technology

[0002] Thanks to the efficient data retrieval capabilities of key-value stores, software applications often use them as a medium for fast caching. However, while the scale and complexity of internet services increase with their expansion and the scope of applications, the response speed and availability of fast caches directly facing end users must not deteriorate accordingly. Therefore, addressing the performance bottlenecks when reading and writing large amounts of cached data is crucial to ensuring user experience and service stability.

[0003] Therefore, the present invention provides a method, system, device, and storage medium for data fragmentation reading and writing in key-value storage. Summary of the Invention

[0004] In view of the problems in the prior art, the purpose of this invention is to provide a method, system, device and storage medium for data fragmentation reading and writing in key-value storage, which overcomes the difficulties of the prior art, can automatically count the details of enterprise commercial software installation, monitor the installation status of risky software, and mark the details of paid software that the enterprise has not purchased licenses for.

[0005] Embodiments of the present invention provide a method for data fragmentation read and write in key-value storage, comprising the following steps:

[0006] S110. Read the original data from each database table of the original data source and process it into an original cache data packet;

[0007] S120. Logically fragment the original cached data packet to obtain several sub-data packets, and write the number of the sub-data packets and the sub-data packets together into the key-value storage;

[0008] S130. When the first data key is read from the key-value store, determine whether it is normal cached data or the number of sub-data packets; and

[0009] S140. When the data key is the number of sub-data packets, start the corresponding number of threads to concurrently read and process all data fragments and aggregate them into the original cached data packets.

[0010] Preferably, step S120 includes:

[0011] S121. Determine whether the data volume of the original cached data packet is greater than a preset threshold. If yes, proceed to step S122; otherwise, proceed to step S124.

[0012] S122. Logically fragment the original cached data packet to obtain several sub-data packets;

[0013] S123, Store the number of sub-data packets and the sub-data packets, and execute step S130; and

[0014] S124. Write data to a single key-value pair and proceed to step S130.

[0015] Preferably, in step S122, the original cached data packet is fragmented and serialized and compressed based on the cached data of the preset sub-data packets, and the number of sub-data packets is obtained.

[0016] Preferably, step S123 includes: writing the number of sub-data packets and the sub-data packets into their respective key-value pairs in the key-value storage, wherein the number of cached data for each sub-data packet is equal, and then executing step S130.

[0017] Preferably, the method further includes step S150: when the data key is binary cache data, the binary cache data is decompressed and deserialized into the original data packet.

[0018] Preferably, step S130 includes:

[0019] Read the first sub-data packet from the key-value store and determine whether it is binary cached data or a number representing the number of sub-data packets;

[0020] If the data in the first sub-data packet is a number representing the number of sub-data packets, then step S140 is executed;

[0021] If the data in the first sub-data packet is binary cached data, then step S150 is executed.

[0022] Preferably, step S140 includes:

[0023] S141. When the data key is the number of sub-data packets, start the corresponding number of threads;

[0024] S142. Each thread concurrently reads the corresponding sub-data packet, decompresses and deserializes it, and finally merges them into the result set;

[0025] S143. After each thread completes its task, the final result set is the original cached data packet.

[0026] Embodiments of the present invention also provide a data fragmentation read / write system for key-value stores, used to implement the above-described data fragmentation read / write method for key-value stores, wherein the data fragmentation read / write system for key-value stores includes:

[0027] The raw caching module reads raw data from various database tables of the raw data source and processes it into raw cache data packets;

[0028] The key-value writing module logically fragments the original cached data packet to obtain several sub-data packets, and writes the number of sub-data packets and the sub-data packets together into the key-value storage;

[0029] The key-value reading module determines, when reading the first data key from the key-value storage, whether it is normal cached data or the number of sub-data packets.

[0030] The data aggregation module, when the data key is the number of sub-data packets, starts the corresponding number of threads to concurrently read and process all data fragments and aggregate them into the original cached data packets.

[0031] Embodiments of the present invention also provide a device for data sharding and reading / writing in a key-value store, comprising:

[0032] processor;

[0033] A memory in which executable instructions of the processor are stored;

[0034] The processor is configured to perform the steps of the above-described method for reading and writing data in key-value storage by executing the executable instructions.

[0035] Embodiments of the present invention also provide a computer-readable storage medium for storing a program, which, when executed, implements the steps of the above-described method for reading and writing data fragments in key-value storage.

[0036] The purpose of this invention is to provide a method, system, device, and storage medium for data sharding and reading / writing in key-value stores. This method can automatically collect statistics on the installation details of commercial software in enterprises, monitor the installation status of potentially risky software, mark the details of paid software that enterprises have not purchased licenses for, provide data support for cleanup work, and automatically update paid versions to the latest free versions. Attached Figure Description

[0037] Other features, objects, and advantages of the invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings.

[0038] Figure 1 This is a flowchart of the data fragmentation read / write method in key-value storage according to the present invention.

[0039] Figure 2 This invention describes the overall read / write process in the data sharding read / write method for key-value storage.

[0040] Figure 3 This is a flowchart of the method for writing cached data in the data sharding read and write of key-value storage according to the present invention.

[0041] Figure 4 This is a flowchart of reading cached data in the data sharding read / write method of key-value storage according to the present invention.

[0042] Figure 5 This is a schematic diagram of the data fragmentation read / write system in key-value storage according to the present invention.

[0043] Figure 6 This is a schematic diagram of the structure of the data fragmentation read / write device in key-value storage according to the present invention.

[0044] Figure 7 This is a schematic diagram of the structure of a computer-readable storage medium according to an embodiment of the present invention. Detailed Implementation

[0045] The following specific examples illustrate the implementation methods of this application. Those skilled in the art can easily understand the other advantages and effects of this application from the content disclosed herein. This application can also be implemented or applied through other different specific embodiments, and various details in this application can be modified or changed according to different viewpoints and application systems without departing from the spirit of this application. It should be noted that, unless otherwise specified, the embodiments and features in the embodiments of this application can be combined with each other.

[0046] The embodiments of this application will now be described in detail with reference to the accompanying drawings, so that those skilled in the art can easily implement the application. This application may be embodied in many different forms and is not limited to the embodiments described herein.

[0047] In this application, the terms "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics represented in connection with that embodiment or example, which are included in at least one embodiment or example of this application. Furthermore, the specific features, structures, materials, or characteristics represented may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate different embodiments or examples represented in this application, as well as features of different embodiments or examples.

[0048] Furthermore, the terms "first" and "second" are used for illustrative purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the representation of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0049] For the purpose of clearly describing this application, devices that are not relevant to the description are omitted, and the same or similar components throughout the specification are given the same reference numerals.

[0050] Throughout this specification, when it is said that a device is "connected" to another device, this includes not only "direct connection" but also "indirect connection" by placing other components in between. Furthermore, when it is said that a device "comprises" a certain constituent element, unless otherwise stated otherwise, this does not exclude other constituent elements, but rather implies that other constituent elements may be included.

[0051] When we say that a device is "above" another device, this can mean that it is directly above the other device, or it can mean that other devices are present in between. Conversely, when we say that a device is "directly" "above" another device, there are no other devices present in between.

[0052] Although the terms first, second, etc., are used in some instances herein to refer to various elements, these elements should not be limited by these terms. These terms are used only to distinguish one element from another. For example, first interface and second interface, etc., are used. Furthermore, as used herein, the singular forms “a,” “an,” and “the” are intended to also include the plural forms unless the context indicates otherwise. It should be further understood that the terms “comprising,” “including,” indicate the presence of features, steps, operations, elements, components, items, kinds, and / or groups, but do not exclude the presence, occurrence, or addition of one or more other features, steps, operations, elements, components, items, kinds, and / or groups. The terms “or” and “and / or” as used herein are interpreted as inclusive, or mean any one or any combination thereof. Thus, “A, B, or C” or “A, B, and / or C” means “any one of: A; B; C; A and B; A and C; B and C; A, B, and C.” Exceptions to this definition will only occur if the combination of elements, functions, steps, or operations is inherently mutually exclusive in some way.

[0053] The technical terms used herein are for reference only to specific embodiments and are not intended to limit the scope of this application. The singular form used herein includes the plural form unless the statement explicitly indicates otherwise. The word "comprising" as used in the specification means to specify a particular characteristic, region, integer, step, operation, element, and / or component, and does not exclude the presence or addition of other characteristics, regions, integers, steps, operations, elements, and / or components.

[0054] Although not explicitly defined, all terms, including technical and scientific terms used herein, shall have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. Terms defined in commonly used dictionaries shall be further interpreted as having a meaning consistent with the relevant technical literature and the content of this present application, and shall not be over-interpreted as having an ideal or overly formulaic meaning unless otherwise defined.

[0055] Figure 1 This is a flowchart of the data fragmentation read / write method in key-value storage according to the present invention. For example... Figure 1 As shown, the method for data fragmentation reading and writing in key-value storage according to the present invention includes:

[0056] S110. Read the original data from each database table of the original data source and process it into an original cache data packet.

[0057] S120. Logically fragment the original cached data packet to obtain several sub-data packets, and write the number of sub-data packets and the sub-data packets together into the key-value store.

[0058] S130. When the first data key is read from the key-value store, determine whether it is normal cached data or the number of sub-data packets.

[0059] S140. When the data key is the number of sub-data packets, start the corresponding number of threads to concurrently read and process all data fragments and aggregate them into the original cached data packets, but this is not a limitation.

[0060] When reading large amounts of cached data, the post-processing steps such as decompression, deserialization, and data verification often take significantly longer than the main processes of data retrieval and network transmission due to the serial nature of the process. This is particularly pronounced when reading large cached data packets with complex post-processing, causing significant load impacts on both the reading and writing applications. Specifically, when the request frequency to the key-value storage cluster reaches millions of times per second, if the cached data size read or written by the software application exceeds hundreds of megabytes, it can cause I / O blocking thousands of times slower than normal read times, leading to a performance avalanche effect. Therefore, this method aims to reduce the performance load on the data source and the reading end when accessing large amounts of cached data in key-value storage, improve reading and post-processing efficiency, and ensure the availability and response speed of the entire software system.

[0061] In a preferred embodiment, step S120 includes:

[0062] S121. Determine whether the amount of data in the original cached data packet is greater than the preset threshold. If yes, proceed to step S122; otherwise, proceed to step S124.

[0063] S122. Logically fragment the original cached data packet to obtain several sub-data packets.

[0064] S123. Store the number of sub-data packets and the sub-data packets, and then execute step S130.

[0065] S124. Write data to a single key-value pair and proceed to step S130, but not limited to this step.

[0066] In a preferred embodiment, in step S122, the original cached data packet is fragmented and serialized and compressed based on the cached data of the preset sub-data packets, and the number of sub-data packets is obtained, but is not limited thereto.

[0067] In a preferred embodiment, step S123 includes: writing the number of sub-data packets and the sub-data packets into their respective key-value pairs in the key-value store, with the number of cached data packets for each sub-data packet being equal, and then performing step S130, but is not limited thereto.

[0068] In a preferred embodiment, the method further includes step S150: if the data key is binary cache data, then the binary cache data is decompressed and deserialized into the original data packet, but this is not a limitation.

[0069] In a preferred embodiment, step S130 includes:

[0070] Read the first sub-data packet from the key-value store and determine whether it is binary cached data or a number representing the number of sub-data packets.

[0071] If the data in the first sub-data packet is a number representing the number of sub-data packets, then proceed to step S140.

[0072] If the data in the first sub-data packet is binary cached data, then step S150 is executed, but this is not the only option.

[0073] In a preferred embodiment, step S140 includes:

[0074] S141. When the data key is the number of sub-data packets, start the corresponding number of threads.

[0075] S142. Each thread concurrently reads the corresponding sub-data packet, decompresses and deserializes it, and finally merges them into the result set.

[0076] S143. After each thread completes its task, the final result set is the original cached data packet, but this is not the only option.

[0077] This invention can statistically analyze the installation details of enterprise commercial software, monitor the installation status of potentially risky software, and mark the details of paid software that the enterprise has not purchased licenses for, providing data support for cleanup efforts. This invention is primarily aimed at software applications that need to frequently and quickly read large amounts of cached data from key-value stores. Specifically, during the write process, if a cache exceeds a certain data size threshold, this method pre-shards it according to business logic, dividing it into several appropriately sized sub-data packets, and writes all sub-data packets and their quantities into multiple key-value pairs in the key-value store. Thus, when a large amount of cache data needs to be read during the read process, the application first obtains the quantity of all sub-data packets, and then can initiate multiple threads concurrently for data retrieval, network transmission, post-processing, and merging of all sub-data packets, ultimately obtaining the entire original cached data.

[0078] The implementation process of this invention is as follows:

[0079] First, the original data is read from each database table of the original data source and processed into original cache data packets.

[0080] Then, it is determined whether the amount of data in the original cached data packet exceeds a preset threshold. If so, the original cached data packet is logically fragmented to obtain several sub-data packets. Based on the cached data of the preset sub-data packets, the original cached data packet is fragmented and serialized and compressed to obtain the number of sub-data packets. The number of sub-data packets and the sub-data packets are written into their respective key-value pairs in the key-value store, with the amount of cached data in each sub-data packet being equal. If not, data is written to a single key-value pair.

[0081] Read the first sub-data packet from the key-value store and determine whether it is binary cached data or a number representing the number of sub-data packets. If the data in the first sub-data packet is a number representing the number of sub-data packets...

[0082] When the data key is the number of sub-data packets, the corresponding number of threads are started. Each thread concurrently reads the corresponding sub-data packets, decompresses and deserializes them, and finally merges them into the result set. After each thread completes its task, the final result set is the original cached data packets.

[0083] When the data key is binary cached data, the binary cached data is decompressed and deserialized into the original data packet.

[0084] First, for cached data with large volumes and complex post-processing procedures, the main bottleneck in reading lies in the serialized post-processing steps such as deserialization and decompression. Therefore, by splitting large cached data into multiple logical fragments, the retrieval, network transmission, post-processing, and merging of each sub-data packet can be performed concurrently during the reading process, thereby significantly improving reading efficiency. Simultaneously, since the size of each sub-data packet is relatively moderate, it does not impose excessive resource consumption on both the reader and writer, reducing the performance load on both parties and improving system stability and availability. Ultimately, this ensures that reading time does not increase exponentially with the size of the data, keeping the response speed and user experience within a controllable range.

[0085] Figure 2 This invention describes the overall read / write process in the data sharding read / write method for key-value storage. Figure 3 This is a flowchart of the method for writing cached data in the data sharding read and write of key-value storage according to the present invention. Figure 4 This is a flowchart illustrating the process of reading cached data in the data sharding read / write method for key-value storage according to the present invention. For example... Figures 2 to 4 As shown, the implementation process of the present invention is as follows:

[0086] refer to Figure 2 Overall read / write process:

[0087] The application reads raw data from various database tables in the original data source and processes it into raw cache data packets.

[0088] The application writes logical fragments of the original data packets based on the data volume, and writes the number of sub-data packets and the sub-data packets together into the key-value store.

[0089] The business application first reads the first data key from the key-value store to determine whether it is normal cached data or the number of sub-data packets.

[0090] If it is the number of sub-data packets, then start the corresponding number of threads to concurrently read and process all data fragments, and aggregate them into the original cached data packets.

[0091] refer to Figure 3 The writing process is as follows: The process of writing cached data begins with the application. First, it is determined whether the amount of cached data exceeds the threshold.

[0092] Data writing process: If the data volume is lower than the threshold, the data will be serialized, compressed, and directly written into a single key-value pair in the key-value store.

[0093] Data sharding write logic: If the data volume exceeds the threshold, the data packet is sharded according to the business logic, that is, split into several appropriately sized sub-data packets. Each sub-data packet contains an even amount of cached data, and the number of sub-data packets is calculated.

[0094] After each sub-data packet is serialized and compressed, along with the number of sub-data packets, it is written into its respective key-value pair in the key-value store according to certain generation rules.

[0095] refer to Figure 4 The reading process is as follows: The process by which the business application begins reading cached data. First, it determines whether to read the first sub-data packet. Then, it determines whether the data in the first sub-data packet is a number representing the number of sub-data packets, or normal binary cached data.

[0096] Data reading process: If it is not a number, the binary cached data is directly decompressed and deserialized into the original data packet.

[0097] Data fragment reading process: If the data is numerical, a corresponding number of threads are started based on the number of sub-data packets. Each thread concurrently reads the corresponding sub-data packets, decompresses and deserializes them, and finally merges them into the result set. After each thread completes its task, the final result set is the original large data packet.

[0098] In this invention, each shard is hashed into its own key-value pairs according to rules, and the number of shards is stored. When resources are sufficient, the read time is linear, and there is no performance impact on either the reader or the reader. Furthermore, it can be applied to caching scenarios with high request frequency, high real-time requirements, and large data volumes.

[0099] Figure 5 This is a schematic diagram of the data sharding read / write system in key-value storage according to the present invention. For example... Figure 5 As shown, embodiments of the present invention also provide a data fragmentation read / write system for key-value stores, used to implement the above-described data fragmentation read / write method for key-value stores. The data fragmentation read / write system 5 for key-value stores includes:

[0100] The raw caching module 51 reads raw data from various database tables of the raw data source and processes it into raw caching data packets.

[0101] The key-value module 52 writes the original cached data packet into logical fragments to obtain several sub-data packets, and writes the number of sub-data packets and the sub-data packets together into the key-value storage.

[0102] The key-value reading module 53 reads the first data key from the key-value storage and determines whether it is normal cached data or the number of sub-data packets.

[0103] The data aggregation module 54 starts a corresponding number of threads to concurrently read and process all data fragments when the data key is the number of sub-data packets, and aggregates them into the original cached data packets.

[0104] In a preferred embodiment, the key-value writing module 52 is configured to determine whether the amount of data in the original cached data packet is greater than a preset threshold. If so, the original cached data packet is logically fragmented to obtain several sub-data packets, and the number of sub-data packets and the sub-data packets are stored. If not, data is written to a single key-value pair.

[0105] In a preferred embodiment, the write key module 52 is further configured to fragment and serialize the original cached data packet based on the cached data of the preset sub-data packet, and obtain the number of sub-data packets, but is not limited thereto.

[0106] In a preferred embodiment, the write key value module 52 is further configured to write the number of sub-data packets and the sub-data packets into their respective key-value pairs in the key value storage, wherein the number of cached data for each sub-data packet is equal, but not limited thereto.

[0107] In a preferred embodiment, a decompression module 55 is also included, which decompresses and deserializes the binary cache data into the original data packet when the data key is binary cache data, but is not limited thereto.

[0108] In a preferred embodiment, the key-value reading module 53 is configured to read the first sub-data packet from the key-value storage and determine whether it is binary cached data or a number representing the number of sub-data packets. The first sub-data packet contains a number representing the number of sub-data packets. The first sub-data packet contains binary cached data, but this is not a limitation.

[0109] In a preferred embodiment, the data aggregation module 54 is configured to start a corresponding number of threads when the data key is the number of sub-data packets. Each thread concurrently reads the corresponding sub-data packets, decompresses and deserializes them, and finally merges them into the result set. After each thread completes its task, the final result set is the original cached data packets, but this is not a limitation.

[0110] The data sharding read / write system for key-value storage of the present invention can automatically count the installation details of enterprise commercial software, monitor the installation status of risky software, mark the details of paid software that the enterprise has not purchased licenses for, provide data support for cleanup work, and automatically update the paid version to the latest free version.

[0111] This invention also provides a device for fragmented reading and writing of data in a key-value store, including a processor and a memory storing executable instructions of the processor. The processor is configured to execute steps of a method for fragmented reading and writing of data in a key-value store by executing the executable instructions.

[0112] As shown above, the data sharding read / write device for key-value storage of this invention can automatically collect statistics on the installation details of enterprise commercial software, monitor the installation status of risky software, mark the details of paid software that the enterprise has not purchased licenses for, provide data support for cleanup work, and automatically update the paid version to the latest free version.

[0113] Those skilled in the art will understand that various aspects of the present invention can be implemented as systems, methods, or program products. Therefore, various aspects of the present invention can be specifically implemented in the following forms: a completely hardware implementation, a completely software implementation (including firmware, microcode, etc.), or a combination of hardware and software aspects, collectively referred to herein as a "circuit," "module," or "platform."

[0114] Figure 6 This is a schematic diagram of the data fragmentation read / write device in key-value storage according to the present invention. See below for reference. Figure 6 To describe an electronic device 600 according to this embodiment of the present invention. Figure 6 The electronic device 600 shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.

[0115] like Figure 6 As shown, the electronic device 600 is presented in the form of a general-purpose computing device. The components of the electronic device 600 may include, but are not limited to: at least one processing unit 610, at least one storage unit 620, a bus 630 connecting different platform components (including storage unit 620 and processing unit 610), a display unit 640, etc.

[0116] The storage unit stores program code, which can be executed by the processing unit 610 to perform the steps described in the above-described section on the electronic prescription transfer processing method according to various exemplary embodiments of the present invention. For example, the processing unit 610 can perform actions such as... Figure 1 The steps are shown in the figure.

[0117] Storage unit 620 may include a readable medium in the form of a volatile storage unit, such as random access memory (RAM) 6201 and / or cache memory 6202, and may further include a read-only memory (ROM) 6203.

[0118] Storage unit 620 may also include a program / utility 6204 having a set (at least one) program module 6205, such program module 6205 including but not limited to: operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0119] Bus 630 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the multiple bus structures.

[0120] Electronic device 600 can also communicate with one or more external devices 700 (e.g., keyboard, pointing device, Bluetooth device, etc.), and with one or more devices that enable a user to interact with electronic device 600, and / or with any device that enables electronic device 600 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 650. Furthermore, electronic device 600 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 660. Network adapter 660 can communicate with other modules of electronic device 600 via bus 630. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage platforms.

[0121] This invention also provides a computer-readable storage medium for storing a program, which, when executed, implements steps of a method for fragmented reading and writing of data in a key-value store. In some possible implementations, various aspects of this invention can also be implemented as a program product comprising program code that, when run on a terminal device, causes the terminal device to perform the steps described in the above-described electronic prescription processing method section of this specification according to various exemplary embodiments of the invention.

[0122] As shown above, the data sharding read / write system for key-value storage of this invention can automatically collect statistics on the installation details of enterprise commercial software, monitor the installation status of risky software, mark the details of paid software that the enterprise has not purchased licenses for, provide data support for cleanup work, and automatically update the paid version to the latest free version.

[0123] Figure 7 This is a schematic diagram of the structure of the computer-readable storage medium of the present invention. (Reference) Figure 7As shown, a program product 800 for implementing the above-described method according to an embodiment of the present invention is described. This product may employ a portable compact disc read-only memory (CD-ROM) and include program code, and may run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto. In this document, the readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0124] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0125] Computer-readable storage media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable storage medium may also be any readable medium other than a readable storage medium that can transmit, propagate, or transfer a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0126] Program code for performing the operations of this invention can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0127] In summary, the purpose of this invention is to provide a method, system, device, and storage medium for data sharding and reading / writing in key-value stores, which can automatically count the installation details of enterprise commercial software, monitor the installation status of risky software, mark the details of paid software that the enterprise has not purchased licenses for, provide data support for cleanup work, and automatically update paid versions to the latest free versions.

[0128] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various simple deductions or substitutions can be made without departing from the concept of the present invention, and all such modifications and substitutions should be considered within the scope of protection of the present invention.

Claims

1. A method for data fragmentation read / write in key-value storage, characterized in that, Includes the following steps: S110. Read the original data from each database table of the original data source and process it into an original cache data packet; S120. Logically fragment the original cached data packet to obtain several sub-data packets, and write the number of sub-data packets and the sub-data packets together into a key-value store, including: S121. Determine whether the data volume of the original cached data packet is greater than a preset threshold. If yes, proceed to step S122; if no, proceed to step S124; S122. Fragment the original cached data packet based on the cached data of the preset sub-data packets and serialize and compress it to obtain the number of sub-data packets; S123. Store the number of sub-data packets and the sub-data packets, and proceed to step S130; S124. Write data to a single key-value pair, and proceed to step S130. S130. When reading the first data key from the key-value store, determine whether it is normal cached data or the number of sub-data packets, including: reading the first sub-data packet from the key-value store and determining whether it is binary cached data or a number representing the number of sub-data packets; if the data in the first sub-data packet is a number representing the number of sub-data packets, then execute step S140; if the data in the first sub-data packet is binary cached data, then execute step S150. S140. When the data key is the number of sub-data packets, start the corresponding number of threads to concurrently read and process all data fragments, and aggregate them into the original cached data packets; and S150. When the data key is binary cached data, the binary cached data is decompressed and deserialized into the original data packet.

2. The method for data fragmentation and read / write in key-value storage as described in claim 1, characterized in that, Step S123 includes: writing the number of sub-data packets and the sub-data packets into their respective key-value pairs in the key-value storage, with the number of cached data for each sub-data packet being equal, and then executing step S130.

3. The method for data fragmentation and read / write in key-value storage as described in claim 1, characterized in that, Step S140 includes: S141. When the data key is the number of sub-data packets, start the corresponding number of threads; S142. Each thread concurrently reads the corresponding sub-data packet, decompresses and deserializes it, and finally merges them into the result set; S143. After each thread completes its task, the final result set is the original cached data packet.

4. A data fragmentation read / write system for key-value stores, used to implement the data fragmentation read / write method for key-value stores as described in claim 1, characterized in that, include: The raw caching module reads raw data from various database tables of the raw data source and processes it into raw caching data packets; The key-value writing module logically fragments the original cached data packet to obtain several sub-data packets, and writes the number of sub-data packets and the sub-data packets together into the key-value storage; The key-value reading module determines, when reading the first data key from the key-value storage, whether it is normal cached data or the number of sub-data packets. The data aggregation module, when the data key is the number of sub-data packets, starts the corresponding number of threads to concurrently read and process all data fragments and aggregate them into the original cached data packets.

5. A device for data fragmentation and reading / writing in key-value storage, characterized in that, include: processor; A memory in which executable instructions of the processor are stored; The processor is configured to perform the steps of the data fragmentation read / write method in key-value storage as described in claim 1 by executing the executable instructions.

6. A computer-readable storage medium for storing a program, characterized in that, When the program is executed by the processor, it implements the steps of the data fragmentation read / write method in key-value storage as described in claim 1.

Images