Data storage method and data storage device for string data

Abstract

This specification provides a data storage method and device for string data. During data storage, for each string in the string dataset to be stored, a pattern dataset is used for pattern matching. Each pattern in the pattern dataset is common string data among multiple strings in the string dataset and is stored in the pattern data storage area of ​​the data storage system. In response to a string containing matching pattern data, proprietary string data (excluding the matching pattern data) is extracted from the string and stored in the proprietary data storage area of ​​the data storage system. An index relationship is formed between the stored proprietary string data and the corresponding pattern data stored in the pattern data storage area. In response to a string not containing matching pattern data, the entire original string data is stored in the proprietary data storage area.

Description

Technical Field

[0001] The embodiments in this specification generally relate to the field of data storage, and more particularly to data storage methods and devices for string data. Background Technology

[0002] With the rapid development of computer and network technologies, people are increasingly using the internet for access and operations, generating ever-growing amounts of data and information online. This data and information needs to be stored (or cached) in a database system for later retrieval and use. A database system enables data sharing and centralized management, including operations such as writing, reading, and modifying data. The data managed by the database system is stored in a data storage system, thus the space and access speed of the data storage system impact the database system's data management operations. As the amount of data to be stored grows larger, improving the data storage efficiency of the data storage system has become an urgent problem to be solved. Summary of the Invention

[0003] This specification provides a data storage method and device for string data. Using this data storage method and device, the efficiency of string data storage can be improved.

[0004] According to one aspect of an embodiment of this specification, a method for storing string data is provided, comprising: performing pattern matching using a pattern dataset for each string data in a string dataset to be stored, to determine whether the string data contains matching pattern data, wherein the pattern dataset is trained using a set of string data samples sampled from the string dataset to be stored, each pattern data is common string data of multiple string data samples, and each pattern data is stored in a pattern data storage area of ​​a data storage system; in response to the string data containing matching pattern data, extracting proprietary string data other than the matching pattern data from the string data, and storing the extracted proprietary string data in a proprietary data storage area of ​​the data storage system, wherein an index relationship is formed between the stored proprietary string data and the corresponding pattern data stored in the pattern data storage area; and in response to the string data not containing matching pattern data, storing the original data of the string data as a whole in the proprietary data storage area.

[0005] Optionally, in one example of the above aspects, the data structure of the stored proprietary string data includes at least an index data field and a proprietary data field, wherein the index data field stores the index relationship used to index the corresponding pattern data, and the proprietary data field is used to store the proprietary string data.

[0006] Optionally, in one example of the above aspects, when the stored proprietary string data includes multiple proprietary string subsequences, for each proprietary string subsequence, the proprietary data field includes a subsequence data length field and a subsequence data body field, wherein the subsequence data length field stores the data length of the proprietary string subsequence, and the subsequence data body field stores the proprietary string subsequence data.

[0007] Optionally, in one example of the above aspects, the data storage system further includes an index data storage area, in which corresponding index data is stored for each string data. The data structure of the stored index data includes a pattern data index field and a proprietary data index field. The pattern data index field stores pattern data index data for indexing the corresponding pattern data, and the proprietary data index field stores proprietary data index data for indexing the corresponding proprietary string data.

[0008] Alternatively, in one example of the above aspects, the pattern dataset is trained based on a hierarchical clustering algorithm using the string data sample set.

[0009] Optionally, in one example of the above aspects, the pattern dataset is trained using the string data sample set based on the following training process: initializing each string data sample in the string data sample set as initial pattern data to generate an initial pattern dataset; performing the following iterative process until the number of trained pattern data reaches a preset value; calculating the pattern data similarity between each pair of pattern data in the current pattern dataset; merging the two pattern data with the highest pattern data similarity to obtain merged pattern data; and replacing the corresponding two pattern data in the current pattern dataset with the merged pattern data to update the pattern dataset.

[0010] Optionally, in one example of the above aspects, the pattern data similarity includes pattern data distance.

[0011] Optionally, in one example of the above aspects, the pattern data distance includes the coding length gain after merging the two pattern data.

[0012] Optionally, in one example of the above aspects, each pattern data is stored in the pattern data storage area after data compression, and / or each proprietary string data is stored in the proprietary data storage area after data compression.

[0013] Optionally, in one example of the above aspects, the data compression method used for each pattern data is determined based on the data structure and data composition of the pattern data, and the data compression method used for each proprietary string data is determined based on the data structure and data composition of the proprietary string data.

[0014] Optionally, in one example of the above aspects, storing the entire original data of the string data in the proprietary data storage area in response to the string data not containing matching pattern data includes: storing the entire original data of the string data in the proprietary data storage area after data compression in response to the string data not containing matching pattern data.

[0015] Optionally, in one example of the above aspects, the data storage system includes multiple data storage devices, and the modal data storage area and the proprietary data storage area are deployed in different data storage devices.

[0016] According to another aspect of the embodiments of this specification, a data storage apparatus for string data is provided, comprising: a pattern data training unit, which trains a pattern dataset using a set of string data samples sampled from the string dataset to be stored, wherein each pattern data is common string data of multiple string data samples; a pattern data storage unit, which stores each pattern data in a pattern data storage area of ​​a data storage system; a pattern matching unit, which performs pattern matching using the pattern dataset for each string data in the string dataset to be stored to determine whether the string data contains matching pattern data; a proprietary data extraction unit, which extracts proprietary string data other than the matching pattern data from each string data in the string dataset to be stored in response to the string data containing matching pattern data; and a proprietary data storage unit, which stores the extracted proprietary string data in the proprietary data storage area of ​​the data storage system for each string data in the string dataset to be stored in response to the string data containing matching pattern data, wherein the stored proprietary string data and the corresponding pattern data stored in the pattern data storage area form an index relationship, and stores the original data of the string data entirely in the proprietary data storage area in response to the string data not containing matching pattern data.

[0017] Optionally, in one example of the above aspects, the data structure of the stored proprietary string data includes at least an index data field and a proprietary data field, wherein the index data field stores the index relationship used to index the corresponding pattern data, and the proprietary data field is used to store the proprietary string data.

[0018] Optionally, in one example of the above aspects, the data storage system further includes an index data storage area, and the data storage device further includes: an index data storage unit, which stores corresponding index data in the index data storage area for each string data, wherein the data structure of the stored index data includes a pattern data index field and a proprietary data index field, wherein the pattern data index field stores pattern data index data for indexing the corresponding pattern data, and the proprietary data index field stores proprietary data index data for indexing the corresponding proprietary string data.

[0019] Alternatively, in one example of the above aspects, the pattern data training unit uses the string data sample set to train the pattern dataset based on a hierarchical clustering algorithm.

[0020] Optionally, in one example of the above aspects, the pattern data training unit includes: a pattern data initialization module, which initializes each string data sample in the string data sample set as initial pattern data to generate an initial pattern dataset; a pattern data similarity calculation module, which calculates the pattern data similarity between each pair of pattern data in the current pattern dataset; a pattern data merging module, which merges the two pattern data with the highest pattern data similarity to obtain merged pattern data; and a pattern dataset update module, which replaces the corresponding two pattern data in the current pattern dataset with the merged pattern data to update the pattern dataset, wherein the pattern data similarity calculation module, the pattern data merging module, and the pattern dataset update module perform a loop process until the number of trained pattern data reaches a preset value.

[0021] Optionally, in one example of the above aspects, the data storage device further includes: a data compression unit for compressing each pattern data and / or each proprietary string data, wherein the pattern data storage unit stores the compressed pattern data in the pattern data storage area, and / or the proprietary data storage unit stores the compressed proprietary string data in the proprietary data storage area.

[0022] Optionally, in one example of the above aspects, in response to the string data not containing matching pattern data, the data compression unit compresses the entire original data of the string data, and the proprietary data storage unit stores the compressed string data in the proprietary data storage area.

[0023] According to another aspect of the embodiments of this specification, a data storage device for string data is provided, comprising: at least one processor, a memory coupled to the at least one processor, and a computer program stored in the memory, wherein the at least one processor executes the computer program to implement the data storage method for string data as described above.

[0024] According to another aspect of the embodiments of this specification, a computer-readable storage medium is provided that stores executable instructions, which, when executed, cause a processor to perform the data storage method for string data as described above.

[0025] According to another aspect of the embodiments of this specification, a computer program product is provided, including a computer program that is executed by a processor to implement the data storage method for string data as described above. Attached Figure Description

[0026] A further understanding of the nature and advantages of this specification can be achieved by referring to the following figures. In the figures, similar components or features may have the same reference numerals.

[0027] Figure 1 A schematic diagram illustrating an application architecture for a data storage system for string data according to an embodiment of this specification is shown.

[0028] Figure 2 An example schematic diagram of string data according to an embodiment of this specification is shown.

[0029] Figure 3 An example flowchart of a data storage method for string data according to an embodiment of this specification is shown.

[0030] Figure 4 An example schematic diagram of a training process for a pattern dataset based on hierarchical clustering, according to an embodiment of this specification, is shown.

[0031] Figure 5 An example flowchart of a pattern dataset training method according to an embodiment of this specification is shown.

[0032] Figure 6 An example schematic diagram is shown illustrating the encoding length gain after merging data with the same pattern data according to an embodiment of this specification.

[0033] Figure 7 An example schematic diagram of stored string data according to an embodiment of this specification is shown.

[0034] Figure 8An example schematic diagram is shown of a data structure for storing proprietary string data that has the same pattern data, according to an embodiment of this specification.

[0035] Figure 9 Another example schematic diagram of stored string data according to an embodiment of this specification is shown.

[0036] Figure 10 An example block diagram of a data storage device according to an embodiment of this specification is shown.

[0037] Figure 11 An example block diagram of a pattern data training unit according to an embodiment of this specification is shown.

[0038] Figure 12 An example schematic diagram of a data storage device based on a computer system according to an embodiment of this specification is shown. Detailed Implementation

[0039] The subject matter described herein will now be discussed with reference to exemplary embodiments. It should be understood that these embodiments are discussed merely to enable those skilled in the art to better understand and implement the subject matter described herein, and are not intended to limit the scope, applicability, or examples set forth in the claims. The function and arrangement of the elements discussed may be changed without departing from the scope of this specification. Various processes or components may be omitted, substituted, or added as needed in the various examples. For example, the described methods may be performed in a different order than described, and steps may be added, omitted, or combined. Furthermore, features described in some examples may be combined in other examples.

[0040] As used herein, the term "comprising" and its variations are open terms meaning "including but not limited to". The term "based on" means "at least partially based on". The terms "one embodiment" and "an embodiment" mean "at least one embodiment". The term "another embodiment" means "at least one other embodiment". The terms "first", "second", etc., may refer to different or the same objects. Other definitions, whether explicit or implicit, may be included below. Unless explicitly indicated by the context, the definition of a term shall remain consistent throughout the specification.

[0041] The flowcharts used in this specification illustrate operations implemented according to some embodiments of this specification. It should be clearly understood that the operations in the flowcharts may not be implemented in a sequential order. Instead, the operations may be implemented in reverse order or simultaneously. Furthermore, one or more additional operations may be added to the flowcharts. One or more operations may be removed from the flowcharts.

[0042] When storing or caching data generated online, it is typically done by storing each data entry as the smallest unit. Furthermore, to improve data storage efficiency and optimize data storage space, a data storage scheme based on data compression has been proposed. In this scheme, data is compressed into fixed-size blocks before storage. The compressed data is then stored in the data storage system. However, this block-based compression scheme does not support random access to the compressed data; accessing a single data entry requires decompressing the entire block, resulting in significant data access latency. Compressing data entry as a single unit will greatly impact the compression ratio, thus affecting data storage efficiency.

[0043] Furthermore, the data generated by online access is string data generated based on specific rules or serialization methods. This string data can be, for example, semi-structured data generated by various rules, such as key-value data, where the key is the value keyword and the value is the corresponding value. Examples of the generated string data include, but are not limited to, JSON data, log data, and XML data. Some strings in this generated string data may have the same substrings. When storing this type of string data, if each data entry is stored as a whole, the same substrings will be stored repeatedly, thus affecting the storage efficiency of the data storage system.

[0044] In view of the above, embodiments of this specification provide a data storage method and device for string data. Using this data storage method and device, a pattern dataset is trained based on a sample set of string data from a string dataset to be stored, and each pattern data in the pattern dataset is stored in a pattern data storage area of ​​a data storage system. Then, for each string data to be stored, pattern matching is performed using the pattern dataset. For string data with matching pattern data, proprietary string data other than the matching pattern data is extracted from the string data, and the extracted proprietary string data is stored in a proprietary data storage area of ​​the data storage system that is independent of the pattern data storage area. An index relationship is formed between the stored proprietary string data and the corresponding pattern data stored in the pattern data storage area. For string data without matching pattern data, the original data of the string data is stored entirely in the proprietary data storage area. Following this method, for multiple strings with the same pattern data, only a single pattern data is stored in the pattern data storage area, and the remaining proprietary string data of each string is stored in the proprietary data storage area, thereby improving the data storage efficiency for string data.

[0045] The data storage method and data storage device according to embodiments of this specification are described below with reference to the accompanying drawings.

[0046] Figure 1 A schematic diagram illustrating an application architecture 100 for a data storage system for string data according to an embodiment of this specification is shown. Figure 1 As shown, the application architecture 100 may include a server 200, a client 300, and a network 120.

[0047] Client 300 may be an electronic device carrying a target application (target APP). User 110 may be a user of client 300. Client 300 may communicate with server 200. In some embodiments, client 300 may have one or more applications (APPs) installed. The installed APPs provide user 110 with the ability and interface to interact with the outside world through network 120. The target APP refers to the client APP corresponding to server 200. In some embodiments, client 300 may include built-in devices or similar content of mobile device 300-1, tablet computer 300-2, laptop computer 300-3, motor vehicle 300-4, or any combination thereof. In some embodiments, mobile device 300-1 may include smart home devices, smart mobile devices, or similar devices, or any combination thereof. In some embodiments, the smart home device may include smart TVs, desktop computers, etc., or any combination thereof. In some embodiments, the smart mobile device may include smartphones, personal digital assistants, etc., or any combination thereof. In some embodiments, client 300 may be a device with positioning technology for locating the position of client 300.

[0048] Server 200 may store data or instructions for performing the data storage method for string data described in the embodiments of this specification, and may execute the data and / or instructions. Client 300 may have access to server 200, thereby accessing the data or instructions stored in server 200 via network 120. Server 200 may include a data storage system for storing string data obtained from client 300. The data storage system may be a memory storage system, a hard disk storage system, or a cache storage system.

[0049] The string data retrieved from client 300 can be string data generated by user 110 when accessing the website online based on certain rules or serialization methods. For example, it could be key-value data, where the key is the value keyword and the value corresponds to that keyword. Examples of the generated string data include, but are not limited to, JSON data, log data, and XML data. Some of these generated string data may share common substrings.

[0050] Figure 2 An example schematic diagram of string data according to an embodiment of this specification is shown. Figure 2In the example string data shown, the first four data entries are business string data generated by the serialization method extracted from the actual database system. The last two data entries are key-value data, each consisting of three key-value pairs. For example, the fifth data entry includes "Name: 'Zhang San'", "Age: '23'", and "Sex: 'Male'". The sixth data entry includes "Name: 'Wang Juan'", "Age: '17'", and "Sex: 'Female'".

[0051] Server 200 can handle all requests for writing, reading, and deleting string data. All requests for writing, reading, and deleting string data are completed by server 200. These requests can originate from client 300. Server 200 can be a standalone server or a distributed server cluster. The distributed server cluster can include multiple distributed sub-servers. These sub-servers can communicate with each other to exchange data and information. For example, the multiple sub-servers can be connected via network 120. The multiple sub-servers can share a common working goal. Each sub-server completes one or more sub-goals of the working goal and passes the results of the sub-goals to other sub-servers that need them. Each sub-server includes one or more processors and may also include one or more storage media (such as memory) to store string data and other data needed to process the sub-goals. Of course, each sub-server can also share one or more storage media (such as memory). The storage media can be connected to each sub-server via network 120. The data or instructions for the key-value storage data compression method described in this specification can be stored in the storage media. The sub-server can be a general-purpose computer or a special-purpose computer. For example, the sub-server can be a server, a personal computer, a portable computer (such as a laptop computer, a tablet computer, etc.), or an electronic device with other computing capabilities.

[0052] Network 120 can facilitate the exchange of information and / or data. For example... Figure 1As shown, client 300 and server 200 can connect to network 120 and transmit information and / or data to each other through network 120. For example, server 200 can obtain string data from client 300 through network 120. In some embodiments, network 120 can be any type of wired or wireless network, or a combination thereof. For example, network 120 may include cable networks, wired networks, fiber optic networks, telecommunications networks, intranets, the Internet, local area networks (LANs), wide area networks (WANs), wireless local area networks (WLANs), metropolitan area networks (MANs), public switched telephone networks (PSTNs), Bluetooth networks, ZigBee networks, near field communication (NFC) networks, or similar networks. In some embodiments, network 120 may include one or more network access points. For example, network 120 may include wired or wireless network access points, and one or more components of client 300 and server 200 can connect to network 120 to exchange data and / or information.

[0053] like Figure 1 As shown, user 110 can perform operations such as writing, deleting, and modifying string data on client 300. These operation requests are transmitted to server 200 via network 120. Server 200 executes the instructions for the string data storage method described in the embodiments of this specification, and stores the written string data in the data storage system.

[0054] Figure 3 An example flowchart of a data storage method 300 for string data according to an embodiment of this specification is shown.

[0055] like Figure 3 As shown, after receiving the string dataset to be stored, operations 310 to 360 are executed in a loop to perform string data storage.

[0056] In execution Figure 3 Before the illustrated iterative process, a pattern dataset needs to be trained using a set of string data samples taken from the string dataset to be stored. The trained pattern dataset can include one or more pattern data points, each of which is a common string extracted from multiple string data samples. For example, based on... Figure 2 The first four strings shown can be used to train a pattern, "VScompany_charging-100". <2,number>accenter <2,number>ac <varchar>counting_log_ <varchar>202 "<6, number>". Then, each extracted pattern data is stored in the pattern data storage area of ​​the data storage system. Here, the pattern data storage area can also be called the shared data storage area or the common data storage area.

[0057] In some embodiments, a pattern dataset is trained based on a hierarchical clustering algorithm using a string data sample set. The string data sample set used may include part or all of the string dataset to be stored.

[0058] Figure 4 A schematic diagram illustrating an example of a training process for a pattern dataset based on hierarchical clustering, according to an embodiment of this specification, is shown. Figure 4 As shown, for the original string data in the string data sample set, the distance between each pair of string data is calculated using minimizing the encoding length as the basic metric. Here, for each string pair, the encoding length gain after merging the two strings is calculated, that is, the increase in encoding length when merging two strings into one string, and the calculated encoding length gain is used as the distance between the two string data in that string pair. Then, a distance matrix (similarity matrix) of the string data sample set is constructed based on the calculated string distances. For example, for n strings, a distance matrix (similarity matrix) can be constructed. A distance matrix of dimension 1 is obtained. Subsequently, hierarchical clustering is performed based on the obtained distance matrix to obtain a preset number of clusters (e.g., k). One pattern data is extracted from each cluster, thus obtaining a pattern dataset with k pattern data.

[0059] Figure 5 An example flowchart of a pattern dataset training method 500 according to an embodiment of this specification is shown.

[0060] like Figure 5 As shown in section 510, each string data sample in the string data sample set is initialized as an initial pattern data to generate an initial pattern dataset. For example, assuming the string data sample set includes n string data samples, each string data sample is treated as an initial pattern data, thereby constructing an initial pattern dataset with n pattern data.

[0061] Then, the training process of the model dataset from 520 to 550 is repeated until the number of trained pattern data reaches a preset value, for example, k.

[0062] Specifically, during each round of model dataset training, at step 520, the pattern similarity between each pair of pattern data in the current pattern dataset is calculated. In some embodiments, pattern similarity may include, for example, pattern distance. In some embodiments, pattern distance may include the encoding length gain after merging two pattern data sets. That is, the increased encoding length after merging two pattern data sets.

[0063] Figure 6 An example schematic diagram is shown illustrating the encoding length gain after merging data with the same pattern data according to an embodiment of this specification.

[0064] like Figure 6 As shown, Figure 2 The first four strings shown are the same pattern data "VScompany_charging-100" in the pattern dataset. <2,number>accenter <2,number>ac <varchar>counting_log_ <varchar>202 "<6,number>" matches. In this pattern data, " "<2,number>" represents a wildcard character that matches numeric data of a fixed length of 2 bytes. <varchar>The "" character represents a wildcard for matching strings of variable length. When calculating the encoding length gain, an additional 1 byte of overhead is required to record the string length.

[0065] When calculating the encoding length gain, for the first string, five proprietary substrings were extracted: "57", "20", "_ac", "Null", and "024050", with encoding length gains of 2 bytes, 2 bytes, 4 bytes, 1 byte, and 6 bytes respectively, totaling 15 bytes. For the second string, five proprietary substrings were extracted: "72", "11", "_ac", "Null", and "204081", with encoding length gains of 2 bytes, 2 bytes, 4 bytes, 1 byte, and 6 bytes respectively, totaling 15 bytes. For the third string, five proprietary substrings were extracted: "15", "42", "Null", "id", and "204420", with encoding length gains of 2 bytes, 2 bytes, 1 byte, 3 bytes, and 6 bytes respectively, totaling 14 bytes. For the fourth string data, five proprietary string subsequences were extracted: "46", "32", "_ac", "Null", and "204081". Their encoding length gains are 2 bytes, 2 bytes, 4 bytes, 1 byte, and 6 bytes, respectively, totaling 15 bytes. Therefore, the total encoding length gain of data with the same pattern after merging is 59 bytes.

[0066] In step 530, the two pattern data with the highest pattern data similarity are merged to obtain the merged pattern data. For example, when pattern data similarity is represented as the encoding length gain after merging two pattern data, the smaller the encoding length gain, the greater the model data similarity.

[0067] At step 540, the merged pattern data is used to replace the corresponding two pattern data points in the current pattern dataset to update the pattern dataset, resulting in an updated pattern dataset. The number of pattern data points in the updated pattern dataset is reduced by one compared to the pattern dataset before the update.

[0068] At step 550, it is determined whether the number of pattern data in the updated pattern dataset has reached a preset number (e.g., k). If the preset number has been reached, the pattern dataset training process ends. If the preset number has not been reached, it returns to step 520, and the updated pattern dataset is used as the current pattern dataset to execute the next training process.

[0069] The following example further illustrates the training process of the model dataset. Assume the string data sample set includes 20 string data samples S1 to S20, with a preset number k of 5. During pattern dataset training, the 20 string data samples S1 to S20 are first initialized as 20 initial pattern data, i.e., C1 to C20. Then, the pattern data similarity between each pair of initial pattern data C1 to C20 is calculated. Assuming the pattern data similarity between C1 and C2 is the highest, pattern data C1 and C2 are merged; that is, the common string portion of pattern data C1 and C2 is extracted as the merged pattern data C1'. Subsequently, the pattern dataset is updated using the merged pattern data C1'; that is, pattern data C1 and C2 are removed from the pattern dataset, and the merged pattern data C1' is added, resulting in updated pattern data with 10 pattern data. Then, using the updated pattern dataset with pattern data C1' and C3-C20 as the current pattern dataset, the process is repeated as above until the updated pattern dataset contains 5 pattern data entries.

[0070] After training the pattern dataset as described above, for each string in the string dataset to be stored, perform operations 310 to 360 in a loop.

[0071] Specifically, in step 310, for the current string data in the string dataset to be stored, pattern matching is performed using the pattern dataset to determine whether the current string data contains matching pattern data. That is, whether the current string data contains a string portion that matches any pattern data in the pattern dataset.

[0072] At 320, in response to the current string data containing matching pattern data, proprietary string data other than the matching pattern data is extracted from the current string data. This proprietary string data can also be referred to as padding sequence data.

[0073] In step 330, the extracted proprietary string data is stored in the proprietary data storage area of ​​the data storage system. An index relationship is formed between the stored proprietary string data and the corresponding pattern data stored in the pattern data storage area. Using this index relationship, the proprietary string data and the corresponding pattern data can be retrieved together during data querying, thus obtaining the complete string data.

[0074] In response to the current string data not containing matching pattern data, at 350, the original data of the current string data is stored entirely in a dedicated data storage area.

[0075] At step 360, check if any unstored string data exists in the string dataset to be stored. If unstored string data exists, return to step 310 and execute the next loop. If no unstored string data exists, the process ends.

[0076] It should be noted that, Figure 3 The examples shown are merely illustrative. In other embodiments, some or all of the string data in the string dataset may be processed in parallel using the above methods.

[0077] In some embodiments, the data storage system may have a schema data storage area and a proprietary data storage area. In this case, the data structure of the stored proprietary string data includes at least an index data field and a proprietary data field. The index data field stores the index relationship used to index the corresponding schema data, and the proprietary data field stores the proprietary string data. In some embodiments, the index relationship can be represented using a schema ID. In some embodiments, the index relationship can be represented using the starting address of the storage location of the corresponding schema data in the schema data storage area. Furthermore, in other embodiments, the index relationship can also be represented using other suitable representation methods.

[0078] Figure 7 An example schematic diagram of stored string data according to an embodiment of this specification is shown. Figure 7 In the example, the string dataset to be stored includes n+1 strings, and the trained pattern dataset includes k patterns. The k pattern data are stored in the pattern data storage area, and each stored pattern data is formed into a data structure with a pattern ID field and a pattern data body field. The pattern ID field stores the pattern ID, and the pattern data body field stores the pattern data body data. For each string data, a corresponding proprietary data is stored in the proprietary data storage area. The stored proprietary data is formed into a data structure with an index data field and a proprietary data field. The index data field stores the pattern ID of the corresponding pattern data, and the proprietary data field stores the extracted proprietary data body.

[0079] In some embodiments, the extracted proprietary data can be further subdivided into multiple proprietary string subsequences. In this case, for each proprietary string subsequence, the proprietary data fields include a subsequence data length field and a subsequence data body field. The subsequence data length field stores the length of the proprietary string subsequence, and the subsequence data body field stores the proprietary string subsequence data.

[0080] For example, targeting Figure 2 The first four strings shown are used to train a pattern data, "VScompany_charging-100". <2,number>accenter <2,number>ac <varchar>counting_log_ <varchar>202 The pattern data, "<6, number>", can be divided into 5 components, each corresponding to a value. The extracted proprietary data is further subdivided into 5 proprietary string subsequences. For example, for... Figure 2 The first data entry contains five proprietary string subsequences: "57", "20", "_ac", "Null", and "024050". Each proprietary string subsequence is stored in a field of the proprietary data's data structure, such as fields Field0, Field1, Field2, Field3, and Field4. Alternatively, in some embodiments, the proprietary data's data structure may also include a corresponding field length field "FieldSize" for each of the fields Field0, Field1, Field2, Field3, and Field4.

[0081] Figure 8 An example schematic diagram is shown of a data structure for storing proprietary string data that has the same pattern data, according to an embodiment of this specification.

[0082] like Figure 8 As shown, based on Figure 2 The first four strings shown are identical to the pattern data "VScompany_charging-100". <2,number>accenter <2,number>ac <varchar>counting_log_ <varchar>202 The string "<6,number>" matches, thus, for the first string, five unique string subsequences are extracted: "57", "20", "_ac", "Null", and "024050". For the second string, five unique string subsequences are extracted: "72", "11", "_ac", "Null", and "204081". For the third string, five unique string subsequences are extracted: "15", "42", "Null", "id", and "204420". For the fourth string, five unique string subsequences are extracted: "46", "32", "_ac", "Null", and "204081". Each unique string subsequence corresponds to a field size FieldSize, thus obtaining... Figure 7 The proprietary data format stored is shown.

[0083] In some embodiments, the data storage system may have a schema data storage area, a proprietary data storage area, and an index data storage area. In this case, for each string of data, corresponding index data is stored in the index data storage area. The data structure of the stored index data includes a schema data index field and a proprietary data index field. The schema data index field stores schema data index data used to index the corresponding schema data, and the proprietary data index field stores proprietary data index data used to index the corresponding proprietary string data.

[0084] Figure 9 An example schematic diagram of stored string data according to an embodiment of this specification is shown. Figure 8 In the example, the string dataset to be stored includes n+1 strings, and the trained pattern dataset includes k patterns. The k pattern data are stored in the pattern data storage area, and each stored pattern data is formed into a data structure with a pattern ID field and a pattern data body field. The pattern ID field stores the pattern ID, and the pattern data body field stores the pattern data body data. For each string data, a corresponding proprietary data is stored in the proprietary data storage area. The stored proprietary data is formed into a data structure with at least a proprietary data field. The proprietary data field stores the extracted proprietary data body. Furthermore, for each string data, a corresponding index data is stored in the index data storage area. The data structure of the stored index data includes a pattern data index field and a proprietary data index field. For example, for the string data identified as id0, the index data maintained in the index data storage area includes a pattern data index "P0" and a proprietary data index "L0". For string data identified as idn, the index data maintained in the index data storage area includes the pattern data index "Pk" and the proprietary data index "L2".

[0085] In some embodiments, each pattern data item may be compressed and stored in a pattern data storage area. The data compression method used for each pattern data item may be the same or different. Optionally, the data compression method used for each pattern data item may be determined based on the data structure and composition of the pattern data. In some embodiments, each proprietary string data item may be compressed and stored in a proprietary data storage area. The data compression method used for each proprietary data item may be the same or different. Optionally, the data compression method used for each proprietary string data item may be determined based on the data structure and composition of the proprietary string data.

[0086] In some embodiments, a data storage system may include multiple data storage devices. Schematic data storage areas and proprietary data storage areas may be deployed in different data storage devices.

[0087] Using the aforementioned data storage method, a pattern dataset is trained based on a sample set of string data from the string dataset to be stored, and each pattern data in the pattern dataset is stored in the pattern data storage area of ​​the data storage system. Then, for each string data to be stored, pattern matching is performed using the pattern dataset. For string data with matching pattern data, proprietary string data other than the matching pattern data is extracted from the string data. The extracted proprietary string data is stored in a proprietary data storage area of ​​the data storage system, independent of the pattern data storage area, and an index relationship is established between the stored proprietary string data and the corresponding pattern data stored in the pattern data storage area. For string data without matching pattern data, the original data of the string data is stored entirely in the proprietary data storage area. Following this method, for multiple strings with the same pattern data, only a single pattern data is stored in the pattern data storage area, while the remaining proprietary string data for each string is stored in the proprietary data storage area, thereby improving the data storage efficiency for string data.

[0088] By using the above data storage method, and by compressing pattern data and / or proprietary data before storing it, the data storage space required to store string data can be further reduced.

[0089] By utilizing the aforementioned data storage methods, and determining suitable data compression methods based on the data structure and composition of pattern data and / or proprietary data, the data compression efficiency of pattern data and / or proprietary data can be further improved, thereby further reducing the data storage space required to store string data.

[0090] Figure 10 An example block diagram of a data storage device 1000 according to an embodiment of this specification is shown. Figure 10 As shown, the data storage device 1000 includes a pattern data training unit 1010, a pattern data storage unit 1020, a pattern matching unit 1030, a proprietary data extraction unit 1040, and a proprietary data storage unit 1050.

[0091] The pattern data training unit 1010 is configured to train a pattern dataset using a set of string data samples sampled from the string dataset to be stored, wherein each pattern data is common string data of multiple string data samples. In some embodiments, the pattern data training unit 1010 may use the set of string data samples to train the pattern dataset based on a hierarchical clustering algorithm.

[0092] Figure 11 An example block diagram of a pattern data training unit 1100 according to an embodiment of this specification is shown. Figure 11 As shown, the pattern data training unit 1100 includes a pattern data initialization module 1110, a pattern data similarity calculation module 1120, a pattern data merging module 1130, and a pattern dataset update module 1140.

[0093] The pattern data initialization module 1110 is configured to initialize each string data sample in the string data sample set as initial pattern data to generate an initial pattern dataset.

[0094] After generating the initial pattern dataset as described above, the pattern data similarity calculation module 1120, the pattern data merging module 1130, and the pattern dataset update module 1140 execute a loop process until the number of trained pattern data reaches a preset value.

[0095] Specifically, in each iteration, the pattern data similarity calculation module 1120 calculates the pairwise pattern data similarity between any two pattern data points in the current pattern dataset. The pattern data merging module 1130 merges the two pattern data points with the highest similarity to obtain merged pattern data. Then, the pattern dataset update module 1140 replaces the corresponding two pattern data points in the current pattern dataset with the merged pattern data to update the pattern dataset. The updated pattern dataset is used as the current pattern dataset for the next iteration.

[0096] The pattern data storage unit 1020 is configured to store each pattern data in the pattern data storage area of ​​the data storage system.

[0097] The pattern matching unit 1030 is configured to perform pattern matching using the pattern dataset for each string data in the string dataset to be stored, in order to determine whether the string data contains matching pattern data.

[0098] The proprietary data extraction unit 1040 is configured to extract proprietary string data, excluding the matched pattern data, from each string data in the string dataset to be stored in response to the string data containing matching pattern data.

[0099] The proprietary data storage unit 1050 is configured to, for each string in the string dataset to be stored, store the extracted proprietary string data in the proprietary data storage area of ​​the data storage system in response to the string data containing matching pattern data. An index relationship is formed between the stored proprietary string data and the corresponding pattern data stored in the pattern data storage area. Furthermore, in response to the string data not containing matching pattern data, the proprietary data storage unit 1050 stores the entire original data of the string data in the proprietary data storage area.

[0100] In some embodiments, the data storage system may have a schema data storage area and a proprietary data storage area. In this case, the data structure of the stored proprietary string data includes at least an index data field and a proprietary data field. The index data field stores the index relationship used to index the corresponding schema data, and the proprietary data field is used to store the proprietary string data.

[0101] In some embodiments, the extracted proprietary data can be further subdivided into multiple proprietary string subsequences. In this case, for each proprietary string subsequence, the proprietary data fields include a subsequence data length field and a subsequence data body field. The subsequence data length field stores the length of the proprietary string subsequence, and the subsequence data body field stores the proprietary string subsequence data.

[0102] In some embodiments, the data storage system may have a pattern data storage area, a proprietary data storage area, and an index data storage area. In this case, the data storage device 1000 may further include an index data storage unit (not shown). For each string of data, the index data storage unit stores corresponding index data in the index data storage area. The data structure of the stored index data includes a pattern data index field and a proprietary data index field. The pattern data index field stores pattern data index data used to index the corresponding pattern data, and the proprietary data index field stores proprietary data index data used to index the corresponding proprietary string data.

[0103] In some embodiments, the data storage device 1000 may further include a data compression unit (not shown). The data compression unit compresses each pattern data and / or each proprietary string data. Subsequently, the pattern data storage unit stores the compressed pattern data in the pattern data storage area, and the proprietary data storage unit stores the compressed proprietary string data in the proprietary data storage area.

[0104] In some embodiments, for each string in the string dataset to be stored, in response to the string not containing matching pattern data, the data compression unit compresses the entire original data of the string. Then, the proprietary data storage unit stores the compressed string data in a proprietary data storage area.

[0105] As per the above reference Figures 1 to 11 This specification describes a data storage method and a data storage device according to embodiments thereof. The data storage device described above can be implemented in hardware, software, or a combination of hardware and software.

[0106] Figure 12 A schematic diagram illustrating an example of a data storage device 1200 implemented on a computer system according to an embodiment of this specification is shown. The data storage device 1200 can be deployed on a server 200, a sub-server of the server 200, or a client 300. The following description uses the data storage device 1200 located on the server 200 as an example. The server 200 can execute the data storage method for string data described in the embodiments of this specification.

[0107] like Figure 12 As shown, the data storage device 1200 includes at least one memory 1230 and at least one processor 1220. In some embodiments, the data storage device 1200 may further include a communication port 1250 and an internal communication bus 1210. Furthermore, the data storage device 1200 may also include I / O components 1260.

[0108] The internal communication bus 1210 can connect different system components, including memory 1230 and processor 1220. I / O component 1260 supports input / output between server 200 and other components.

[0109] The memory 1230 may include non-transitory memory or temporary memory. For example, the memory 1230 may include one or more of a disk 1232, a read-only storage medium (ROM) 1234, or a random access storage medium (RAM) 1236. The memory 1230 may also store at least one instruction set. The instructions are computer program code, which may include programs, routines, objects, components, data structures, procedures, modules, etc., that execute the data storage method for string data provided in the embodiments of this specification.

[0110] Communication port 1250 is used for data communication between data storage device 1200 and external sources. For example, data storage device 1200 can connect to network 120 via communication port 1250. At least one processor 1220 is communicatively connected to at least one memory 1230 via an internal communication bus 1210. At least one processor 1220 is used to execute at least one instruction set as described above. When data storage is performed, at least one processor 1220 reads at least one instruction set stored in memory 1230 and executes the data storage method for string data provided in the embodiments of this specification according to the instructions of at least one instruction set. Processor 1220 can execute all the steps included in the data storage method for string data.

[0111] Processor 1220 may be in the form of one or more processors. In some embodiments, processor 1220 may include one or more hardware processors, such as microcontrollers, microprocessors, reduced instruction set computers (RISC), application-specific integrated circuits (ASICs), application-specific instruction set processors (ASIPs), central processing units (CPUs), graphics processing units (GPUs), physical processing units (PPUs), microcontroller units, digital signal processors (DSPs), field-programmable gate arrays (FPGAs), advanced RISC machines (ARMs), programmable logic devices (PLDs), any circuitry or processor capable of performing one or more functions, etc., or any combination thereof. This is merely for illustrative purposes. Figure 12 The illustrated data storage device 1200 describes only one processor 1220. However, it should be noted that in other embodiments, the data storage device 1200 may also include multiple processors. Therefore, the operation and / or method steps disclosed in the embodiments of this specification may be executed by one processor as described herein, or they may be executed jointly by multiple processors. For example, if steps A and B are executed by processor 1220 of data storage device 1200, it should be understood that steps A and B may also be executed jointly or separately by two different processors 1220 (e.g., a first processor executes step A, a second processor executes step B, or the first and second processors jointly execute steps A and B). Although the above structure describes server-side 200, this structure also applies to client-side 300.

[0112] It should be understood that the computer-executable instructions stored in memory, when executed, cause at least one processor 1220 to perform the above-described embodiments of this specification. Figures 1-11 The description includes various operations and functions.

[0113] Another aspect of embodiments of this specification provides a computer-readable storage medium (e.g., a non-transitory machine-readable medium) storing at least one set of executable instructions for performing string data storage, which, when executed by a processor, instruct the processor to implement the steps of the data storage method for string data described in embodiments of this specification. In some possible implementations, the various aspects provided in embodiments of this specification can also be implemented as a program product comprising program code. When the program product is run on a data storage device provided according to embodiments of this specification, the program code causes the data storage device to perform the steps of the data storage method for string data described in embodiments of this specification. A program product for implementing the above method may employ a portable compact disk read-only memory (CD-ROM) and include program code, and may run on the data storage device provided in embodiments of this specification. However, the program product provided in embodiments of this specification is not limited thereto. In this specification, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system (e.g., processor 1220). The program product may employ any combination of one or more readable media. A computer-readable storage medium may be a readable signal medium or a readable storage medium. Computer-readable storage media can be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media can include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. The computer-readable storage medium can include a data signal propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable storage medium can also be any computer-readable medium other than a computer-readable storage medium that can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable storage medium can be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.Program code for performing the operations provided in the embodiments of this specification can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. Alternatively, the program code can be downloaded from a server computer or the cloud via a communication network.

[0114] Those skilled in the art will understand that the various embodiments disclosed above can be modified and varied without departing from the spirit of the invention. Therefore, the scope of protection of this invention should be defined by the appended claims.

[0115] It should be noted that not all steps and units in the above process and system structure diagrams are mandatory; some steps or units can be omitted according to actual needs. The execution order of each step is not fixed and can be determined as needed. The device structure described in the above embodiments can be a physical structure or a logical structure; that is, some units may be implemented by the same physical entity, or some units may be implemented by multiple physical entities, or they may be jointly implemented by certain components in multiple independent devices.

[0116] In the above embodiments, the hardware units or modules can be implemented mechanically or electrically. For example, a hardware unit, module, or processor may include permanent dedicated circuitry or logic (such as a dedicated processor, FPGA, or ASIC) to perform the corresponding operation. The hardware unit or processor may also include programmable logic or circuitry (such as a general-purpose processor or other programmable processor), which can be temporarily configured by software to perform the corresponding operation. The specific implementation method (mechanical, dedicated permanent circuitry, or temporarily configured circuitry) can be determined based on cost and time considerations.

[0117] The specific embodiments described above with reference to the accompanying drawings are exemplary embodiments, but do not represent all embodiments that can be implemented or fall within the scope of the claims. The term "exemplary" as used throughout this specification means "serving as an example, instance, or illustration" and does not imply that it is "preferred" or "advantageous" compared to other embodiments. Specific details are included to provide an understanding of the described techniques. However, these techniques can be practiced without these specific details. In some instances, well-known structures and apparatuses are shown in block diagram form to avoid obscuring the concepts of the described embodiments.

[0118] The foregoing description of this disclosure is provided to enable any person skilled in the art to implement or use this disclosure. Various modifications to this disclosure will be apparent to those skilled in the art, and the general principles defined herein can be applied to other variations without departing from the scope of this disclosure. Therefore, this disclosure is not limited to the examples and designs described herein, but is consistent with the widest scope of the principles and novel features disclosed herein.< / varchar> < / varchar> < / varchar> < / varchar> < / varchar> < / varchar> < / varchar> < / varchar> < / varchar>

Claims

1. A method for storing string data, comprising: For each string in the string dataset to be stored, Pattern matching is performed using a pattern dataset to determine whether the string data contains matching pattern data. The pattern dataset is trained using a set of string data samples sampled from the string dataset to be stored. Each pattern data is the common string data of multiple string data samples, and each pattern data is stored in the pattern data storage area of ​​the data storage system. In response to the string data containing matching pattern data, proprietary string data other than the matching pattern data is extracted from the string data, and the extracted proprietary string data is stored in the proprietary data storage area of ​​the data storage system. An index relationship is formed between the stored proprietary string data and the corresponding pattern data stored in the pattern data storage area. as well as In response to the string data not containing matching pattern data, the original data of the string data is stored entirely in the dedicated data storage area.

2. The data storage method as described in claim 1, wherein, The data structure of the stored proprietary string data includes at least an index data field and a proprietary data field. The index data field stores the index relationship used to index the corresponding pattern data, and the proprietary data field is used to store the proprietary string data.

3. The data storage method as described in claim 2, wherein, When the stored proprietary string data includes multiple proprietary string subsequences, for each proprietary string subsequence, the proprietary data field includes a subsequence data length field and a subsequence data body field. The subsequence data length field stores the data length of the proprietary string subsequence, and the subsequence data body field stores the proprietary string subsequence data.

4. The data storage method as described in claim 1, wherein, The data storage system also has an index data storage area. For each string data, corresponding index data is stored in the index data storage area. The data structure of the stored index data includes a pattern data index field and a proprietary data index field. The pattern data index field stores pattern data index data used to index the corresponding pattern data, and the proprietary data index field stores proprietary data index data used to index the corresponding proprietary string data.

5. The data storage method as described in claim 1, wherein, The pattern dataset is trained using the string data sample set based on a hierarchical clustering algorithm.

6. The data storage method as described in claim 5, wherein, The pattern dataset was trained using the string data sample set based on the following training process: Each string data sample in the string data sample set is initialized as an initial pattern data to generate an initial pattern dataset; Execute the following iterative process until the number of trained pattern data reaches a preset value. Calculate the pairwise similarity between pattern data in the current pattern dataset; The two pattern data with the highest pattern data similarity are merged to obtain the merged pattern data; as well as The merged pattern data is used to replace the corresponding two pattern data in the current pattern dataset to update the pattern dataset.

7. The data storage method as described in claim 6, wherein, The pattern data similarity includes pattern data distance.

8. The data storage method as described in claim 7, wherein, The pattern data distance includes the encoding length gain after merging two pattern data.

9. The data storage method as described in claim 1, wherein, Each pattern data is compressed and stored in the pattern data storage area, and / or each proprietary string data is compressed and stored in the proprietary data storage area.

10. The data storage method as described in claim 1, wherein, The data compression method used for each pattern data is determined based on the data structure and composition of that pattern data, and the data compression method used for each proprietary string data is determined based on the data structure and composition of that proprietary string data.

11. The data storage method as described in claim 1, wherein, In response to the string data not containing matching pattern data, storing the entire original data of the string data in the dedicated data storage area includes: In response to the string data not containing matching pattern data, the original data of the string data is compressed and stored in the dedicated data storage area.

12. The method of claim 1, wherein, The data storage system includes multiple data storage devices, and the modal data storage area and the dedicated data storage area are deployed in different data storage devices.

13. A data storage device for string data, comprising: The pattern data training unit trains the pattern dataset using a set of string data samples sampled from the string dataset to be stored. Each pattern data is a common string data of multiple string data samples. The schema data storage unit stores each schema data in the schema data storage area of ​​the data storage system. The pattern matching unit performs pattern matching on each string in the string dataset to be stored, using the pattern dataset to determine whether the string contains matching pattern data. The proprietary data extraction unit extracts proprietary string data from each string in the string dataset to be stored, excluding the matched pattern data, in response to the string data containing matching pattern data. as well as A dedicated data storage unit, for each string in the string dataset to be stored, in response to the string containing matching pattern data, stores the extracted dedicated string data in the dedicated data storage area of ​​the data storage system, and forms an index relationship between the stored dedicated string data and the corresponding pattern data stored in the pattern data storage area; and in response to the string not containing matching pattern data, stores the original data of the string in its entirety in the dedicated data storage area.

14. The data storage device as claimed in claim 13, wherein, The data structure of the stored proprietary string data includes at least an index data field and a proprietary data field. The index data field stores the index relationship used to index the corresponding pattern data, and the proprietary data field is used to store the proprietary string data.

15. The data storage device as claimed in claim 13, wherein, The data storage system also has an index data storage area, and the data storage device further includes: The index data storage unit stores corresponding index data in the index data storage area for each string data. The data structure of the stored index data includes a pattern data index field and a proprietary data index field. The pattern data index field stores pattern data index data used to index the corresponding pattern data, and the proprietary data index field stores proprietary data index data used to index the corresponding proprietary string data.

16. The data storage device as claimed in claim 13, wherein, The pattern data training unit uses the string data sample set to train the pattern dataset based on a hierarchical clustering algorithm.

17. The data storage device of claim 16, wherein, The pattern data training unit includes: The pattern data initialization module initializes each string data sample in the string data sample set as initial pattern data to generate an initial pattern dataset. The pattern data similarity calculation module calculates the pairwise pattern data similarity between any two pattern data points in the current pattern dataset. The pattern data merging module merges the two pattern data sets with the highest similarity to obtain merged pattern data; and The pattern dataset update module replaces the corresponding two pattern data entries in the current pattern dataset with the merged pattern data to update the pattern dataset. The pattern data similarity calculation module, the pattern data merging module, and the pattern dataset update module perform a loop process until the number of trained pattern data reaches a preset value.

18. The data storage device of claim 13, further comprising: The data compression unit compresses each pattern data and / or each proprietary string data. The pattern data storage unit stores the compressed pattern data in the pattern data storage area, and / or the proprietary data storage unit stores the compressed proprietary string data in the proprietary data storage area.

19. The data storage device as claimed in claim 18, wherein, In response to the string data not containing matching pattern data, the data compression unit compresses the entire original data of the string data, and the proprietary data storage unit stores the compressed string data in the proprietary data storage area.

20. A data storage device for string data, comprising: At least one processor, Memory coupled to the at least one processor, and A computer program stored in the memory, which is executed by the at least one processor to implement the data storage method for string data as described in any one of claims 1 to 12.

21. A computer-readable storage medium storing executable instructions that, when executed, cause a processor to perform a data storage method for string data as described in any one of claims 1 to 12.

22. A computer program product comprising a computer program that is executed by a processor to implement the data storage method for string data as described in any one of claims 1 to 12.

Images