Automatic and flexible pattern detection and migration

Abstract

This invention is entitled "Automatic Flexible Pattern Detection and Migration." It discloses a method and apparatus for automatic pattern detection and migration. In one embodiment, a file comprising NoSQL data is received, and one or more data types are detected in a hierarchical data table description. Within records of the NoSQL data, which may be stored in JSON format, field names and their data types are stored as a schema describing the data table. When additional records are parsed, the schema is updated to include additional field names and data types, and the schema may include specifications for some fields, such as repetition and optionality. In one embodiment, the schema is a serialized data format, such as Google Protocol Buffers (Protobuf).

Description

[0001] Cross-references to related applications

[0002] This application claims the benefit of U.S. Provisional Patent Application Serial No. 63 / 113,087, filed November 12, 2020, which is incorporated herein by reference. Background Technology Technical Field

[0004] The implementation scheme disclosed herein relates generally to computational storage, and more specifically, to the creation and modification of existing database schemas.

[0005] Description of related fields

[0006] Databases are typically implemented on a host system, which includes a dedicated processor, memory, and storage devices. When performing database operations (such as select, insert, update, or delete), the application provides instructions to the host system, which then retrieves data from the storage devices and performs the requested operation in memory. The results of the operation are then provided to the application and / or written to the storage devices as needed.

[0007] Developers typically haven't yet sought to implement database operations on data storage devices. These devices are capable of storing massive amounts of data and can scale using processing power and fast local storage. However, the effort required to build a database to operate on a typical data storage device would be extremely resource-intensive and require years of development time. On the other hand, building a data storage device in a generalized way to manage SQL and non-SQL databases would be similarly resource- and time-intensive for development.

[0008] What is needed is a system and method to overcome the shortcomings of existing methods in data storage devices to manage SQL databases and perform database operations on non-SQL databases. Summary of the Invention

[0009] This disclosure relates in its entirety to systems, methods, and apparatuses for automated pattern detection and migration. In one embodiment, a file comprising NoSQL data is received, and one or more data types are detected in a hierarchical data table description. Within records of NoSQL data, which may be stored in JSON format, field names and their data types are stored as a schema describing the data table. When additional records are parsed, the schema is updated to include additional field names and data types, and the schema may include specifications for some fields, such as repetition and optionality. In another embodiment, the schema is a serialized data format, such as Google Protocol Buffers (Protobuf).

[0010] In one embodiment, a data storage device is disclosed, comprising: one or more memory modules; and a controller including a processor configured to perform a method for data pattern detection and migration. In this embodiment, the method includes: receiving a portion of a file comprising schema-less data; identifying one of field names, field types, repeating fields, and optional fields in the schema-less data; creating a schema for the schema-less data including the fields; and storing the schema and data from the file in the one or more memory modules.

[0011] In another embodiment, a controller for a data storage device is disclosed, the controller including: I / O for one or more memory devices; and a processor configured to perform a method for data pattern detection and migration. In this embodiment, the method includes: receiving a portion of a file including a plurality of field delimiters for document-based data; detecting the data type of one of the plurality of field delimiters for the document-based data; defining a pattern for the one of the field delimiters for the document-based data based on the data type; and storing the file and the pattern.

[0012] In another embodiment, a system for storing data is disclosed, the system comprising: one or more memory devices; and an SSD controller device configured to perform methods for data pattern detection and migration. In this embodiment, the method includes: detecting a field hierarchy in a portion of a document, the document comprising one of data including fields based on a structured or semi-structured document, data associated with the fields, and a data type associated with the data; defining a schema based on the field hierarchy, the fields, the data, and the data type; and defining a data table based on the schema and the document. Attached Figure Description

[0013] Therefore, a detailed understanding of the foregoing features of this disclosure, a more specific description of this disclosure, and the foregoing brief overview can be obtained by referring to the embodiments, some of which are shown in the accompanying drawings. However, it should be noted that the drawings only illustrate typical embodiments of this disclosure and should therefore not be considered as limiting its scope, as this disclosure allows for other equally effective embodiments.

[0014] Figure 1 This is a schematic block diagram illustrating a storage system according to the disclosed embodiment, wherein the data storage device can be used as a data storage device for a host device.

[0015] Figure 2This is a schematic block diagram illustrating a database server system according to the disclosed implementation scheme.

[0016] Figure 3 This is a schematic block diagram illustrating an improved data storage device according to the disclosed embodiments.

[0017] Figure 4 The timing of database selection, updating, and insertion according to the disclosed implementation scheme is illustrated.

[0018] Figures 5A to 5D It is a graph showing the performance of various modes according to the disclosed implementation scheme.

[0019] Figure 6 This is a flowchart illustrating a method for creating a schema for schema-less data according to the disclosed implementation scheme.

[0020] Figure 7 This is a flowchart illustrating an automatic pattern detection and migration method according to the disclosed implementation.

[0021] Figures 8A to 8B This is an example of a document based on the disclosed implementation scheme.

[0022] For ease of understanding, the same reference numerals are used where possible to denote the same elements common in the accompanying drawings. It is conceivable that elements disclosed in one embodiment may be advantageously used in other embodiments without specific description. Detailed Implementation

[0023] In the following text, reference is made to embodiments of this disclosure. However, it should be understood that this disclosure is not limited to the specifically described embodiments. Rather, consider any combination of the following features and elements (whether or not related to different embodiments) to implement and practice this disclosure. Furthermore, while embodiments of this disclosure may achieve advantages over other possible solutions and / or over the prior art, achieving a particular advantage through a given embodiment is not a limitation of this disclosure. Therefore, the following aspects, features, and embodiments are illustrative only and should not be considered elements or limitations of the appended claims unless expressly stated in the claims. Similarly, reference to “this disclosure” should not be construed as a generalization of any inventive subject matter disclosed herein and should not be considered elements or limitations of the appended claims unless expressly stated in the claims.

[0024] This disclosure relates in its entirety to systems, methods, and apparatuses for automated pattern detection and migration. In one embodiment, a file comprising NoSQL data is received, and one or more data types are detected in a hierarchical data table description. Within records of NoSQL data, which may be stored in JSON format, field names and their data types are stored as a schema describing the data table. When additional records are parsed, the schema is updated to include additional field names and data types, and the schema may include specifications for some fields, such as repetition and optionality. In another embodiment, the schema is a serialized data format, such as Google Protocol Buffers (Protobuf).

[0025] Figure 1 This is a schematic block diagram illustrating a storage system 100 according to the disclosed embodiment, wherein data storage device 106 can be used as a storage device for host device 104. For example, host device 104 can utilize non-volatile (NVM) memory 110 included in data storage device 106 to store and retrieve data. Host device 104 includes host DRAM 138. In some examples, storage system 100 may include multiple storage devices, such as data storage device 106, that can operate as a storage array. For example, storage system 100 may include multiple data storage devices 106 configured to collectively serve as a low-cost / independent disk (RAID) redundant array of high-capacity storage devices for host device 104.

[0026] Storage system 100 includes a host device 104 that can store data to and / or retrieve data from one or more storage devices, such as data storage device 106. Figure 1 As shown, host device 104 can communicate with data storage device 106 via interface 114. Host device 104 may include any of a variety of devices, including computer servers, network attached storage (NAS) units, desktop computers, laptops, tablets, set-top boxes, mobile phones such as so-called "smart" phones, so-called "smart" tablets, televisions, cameras, display devices, digital media players, video game consoles, video streaming devices, or other devices capable of sending or retrieving data from data storage devices.

[0027] Data storage device 106 includes a controller 108, an NVM 110, a power supply 111, volatile memory 112, an interface 114, and a write buffer 116. In some examples, for clarity, data storage device 106 may include... Figure 1Additional components not shown. For example, data storage device 106 may include a printed circuit board (PCB) to which components of data storage device 106 are mechanically attached, and the PCB includes conductive traces for electrically interconnecting components of data storage device 106, etc. In some examples, the physical dimensions and connector configuration of data storage device 106 may conform to one or more standard form factors. Some exemplary standard form factors include, but are not limited to, 3.5” data storage devices (e.g., HDDs or SSDs), 2.5” data storage devices, 1.8” data storage devices, Peripheral Component Interconnect (PCI), PCI Expansion (PCI-X), PCI Express (PCIe) (e.g., PCIe x1, x4, x8, x16, PCIe Mini cards, MiniPCI, etc.). In some examples, data storage device 106 may be directly coupled (e.g., directly soldered) to the motherboard of host device 104.

[0028] The interface 114 of the data storage device 106 may include one or both of a data bus for exchanging data with the host device 104 and a control bus for exchanging commands with the host device 104. The interface 114 may operate according to any suitable protocol. For example, the interface 114 may operate according to one or more of the following protocols: Advanced Technology Attachment (ATA) (e.g., Serial ATA (SATA) and Parallel ATA (PATA)), Fibre Channel Protocol (FCP), Small Computer System Interface (SCSI), Serial Attached SCSI (SAS), PCI and PCIe, Non-Volatile Memory Express (NVMe), OpenCAPI, GenZ, Cache Coherent Interface Accelerator (CCIX), Open Channel SSD (OCSSD), etc. Electrical connections (e.g., the data bus, the control bus, or both) of the interface 114 are electrically connected to the controller 108, thereby providing an electrical connection between the host device 104 and the controller 108, allowing data exchange between the host device 104 and the controller 108. In some examples, the electrical connection of the interface 114 may also allow the data storage device 106 to receive power from the host device 104. For example, such as Figure 1 As shown, power supply 111 can receive power from host device 104 via interface 114.

[0029] NVM 110 may include multiple memory devices or memory cells. NVM 110 may be configured to store and / or retrieve data. For example, a memory cell of NVM 110 may receive data and receive a message from controller 108 instructing the memory cell to store data. Similarly, a memory cell of NVM 110 may receive a message from controller 108 instructing the memory cell to retrieve data. In some examples, each memory cell in the memory cell may be referred to as a die. In some examples, a single physical chip may include multiple dies (i.e., multiple memory cells). In some examples, each memory cell may be configured to store a relatively large amount of data (e.g., 128MB, 256MB, 512MB, 1GB, 2GB, 4GB, 8GB, 16GB, 32GB, 64GB, 128GB, 256GB, 512GB, 1TB, etc.).

[0030] In some examples, each memory cell of the NVM 110 may include any type of non-volatile memory device, such as flash memory device, phase-change memory (PCM) device, resistive random access memory (ReRAM) device, magnetoresistive random access memory (MRAM) device, ferroelectric random access memory (F-RAM), holographic memory device, and any other type of non-volatile memory device.

[0031] NVM 110 may include multiple flash memory devices or memory cells. The NVM flash memory devices may include NAND- or NOR-based flash memory devices and may store data based on the charge contained in the floating gate of the transistors for each flash memory cell. In an NVM flash memory device, the flash memory device may be divided into multiple dies, each of which includes multiple blocks, which may be further divided into multiple pages. Each of the multiple blocks within a particular memory device may include multiple NVM cells. Rows of NVM cells may be electrically connected using word lines to define pages within the multiple pages. Respective cells in each page of the multiple pages may be electrically connected to corresponding bit lines. Furthermore, the NVM flash memory device may be a 2D or 3D device and may be a single-level cell (SLC), multi-level cell (MLC), three-level cell (TLC), or four-level cell (QLC). Controller 108 may write data to and read data from the NVM flash memory device at the page level and erase data from the NVM flash memory device at the block level.

[0032] Data storage device 106 includes a power supply 111 that can provide power to one or more components of the data storage device 106. When operating in standard mode, power supply 111 can use power provided by an external device such as host device 104 to power one or more components. For example, power supply 111 can use power received from host device 104 via interface 114 to power one or more components. In some examples, power supply 111 may include one or more power storage components configured to power one or more components when operating in a shutdown mode, such as when power reception from external devices is stopped. In this way, power supply 111 can be used as an onboard backup power source. Some examples of one or more power storage components include, but are not limited to, capacitors, supercapacitors, batteries, etc. In some examples, the amount of electricity that can be stored by one or more power storage components can be a function of the cost and / or size (e.g., area / volume) of one or more power storage components. In other words, as the amount of electricity stored by one or more power storage components increases, the cost and / or size of one or more power storage components also increases.

[0033] Data storage device 106 also includes volatile memory 112, which can be used by controller 108 to store information. Volatile memory 112 may include one or more volatile memory devices. In some examples, controller 108 may use volatile memory 112 as a cache. For example, controller 108 may store cached information in volatile memory 112 until the cached information is written to non-volatile memory 110. Figure 1 As shown, volatile memory 112 can consume power received from power supply 111. Examples of volatile memory 112 include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static RAM (SRAM), and synchronous dynamic RAM (SDRAM (e.g., DDR1, DDR2, DDR3, DDR3L, LPDDR3, DDR4, LPDDR4, etc.)).

[0034] Data storage device 106 includes a controller 108 that can manage one or more operations of data storage device 106. For example, controller 108 can manage reading data from NVM 110 and / or writing data to NVM 110. In some embodiments, when data storage device 106 receives a write command from host device 104, controller 108 can initiate a data storage command to store data to NVM 110 and monitor the progress of the data storage command. Controller 108 can determine at least one operational characteristic of storage system 100 and store at least one operational characteristic to NVM 110. In some embodiments, when data storage device 106 receives a write command from host device 104, controller 108 temporarily stores the data associated with the write command in internal memory or write buffer 116 before sending the data to NVM 110.

[0035] Figure 2 This is a schematic block diagram illustrating a database server system 200 according to the disclosed embodiment. The database server system includes one or more host devices 202a-202n, wherein each of the one or more host devices 202a-202n may be Figure 1 The system includes host device 104, cloud network 204, network switch 206, and one or more network storage systems 210a-210n. Each of the network storage systems 210a-210n includes one or more data storage devices 212a-212n, wherein each of the one or more data storage devices 212a-212n may be... Figure 1 Data storage device 106 or Figure 3 The data storage device 304 is discussed below.

[0036] One or more host devices 202a-202n can be connected to cloud network 204 via network data transmission methods (such as Ethernet, Wi-Fi, etc.). Cloud network 204 is connected to network switch 206 via network data transmission methods (such as Ethernet, Wi-Fi, etc.). Network switch 206 can resolve incoming and outgoing data to the relevant location. Network switch 206 is coupled to one or more network storage systems 210a-210n. Data from one or more host devices 202a-202n is stored in at least one of one or more data storage devices 212a-212n of the one or more network storage devices 210a-210n.

[0037] For example, one or more network storage systems may be configured to further parse incoming data to one or more corresponding data storage devices 212a-212n, and retrieve data stored at the corresponding one or more data storage devices 212a-212n to be sent to one or more host devices 202a-202n. One or more host devices 202a-202n may be configured to upload and / or download data via cloud network 204, wherein data is uploaded and / or stored to at least one of the one or more data storage devices 212a-212n of one or more network storage systems 210a-210n. It should be understood that “n” refers to the maximum number of components of the database server system 200. For example, one or more data storage devices 212a-212n may be about one data storage device, about two data storage devices, or any number of data storage devices greater than about two.

[0038] Figure 3 This is a schematic block diagram illustrating a storage system 300 of an improved data storage device 304 according to the disclosed embodiment. The storage system 300 may be... Figure 1 The database server system 200. For example, data storage device 304 may be implemented as one or more data storage devices 212a-212n of one or more network storage systems 210a-210n, and host device 302 may be implemented as Figure 2 One or more host devices 202a-202n. It should be understood that, for clarity, data storage device 304 may include... Figure 3 Additional components not shown. In one embodiment, data storage device 304 may be an E1.L enterprise and data SSD form factor (EDSFF).

[0039] Data storage device 304 includes a front-end (FE) application-specific integrated circuit (ASIC) 306, a first front-end module (FM) ASIC 310a, and an nth FM ASIC 310n. In the embodiments described herein, "n" refers to the maximum number of said components in data storage system 304. For example, data storage device 304 may include about 10 FM ASICs, wherein the nth or "n"th number of FM ASICs is equal to about 10. Data storage device 304 also includes one or more NVM dies 316a-316n, 322a-322n. Furthermore, data storage device 304 may include a plurality of FM ASICs (indicated by ellipses), wherein each of the plurality of FM ASICs is coupled to a corresponding NVM die among the plurality of NVM dies 316a-316n, 322a-322n. It should be understood that although multiple FM ASICs and multiple NVM dies coupled to each of the multiple FM ASICs are described, data storage device 304 may include a single FM ASIC coupled to a single NVM die or a single FM ASIC coupled to multiple NVM dies. In one embodiment, the NVM is a NAND memory, wherein each of the multiple NVM dies is a NAND die. In one embodiment, the multiple NVM dies 316a-316n, 322a-322n of data storage device 304 are Bit Cost Scalable (BiCS) 6NVM dies. Compared to previous versions such as BiCS 5NVM dies, BiCS 6NVM dies can have improved operating speed and lower power consumption.

[0040] Each of the multiple FM ASICs 310a-310n includes multiple Low-Density Parity-Check (LDPC) engines 312a-312n, 318a-318n and multiple Flash Interface Modules (FIMs) 314a-314n, 320a-320n. Each of the multiple FIMs 314a-314n, 320a-320n is coupled to a corresponding NVM die among a multiple NVM dies 316a-316n, 322a-322n. In one embodiment, each FIM is coupled to a corresponding NVM die. In another embodiment, each FIM is coupled to approximately four corresponding NVM dies. The multiple LDPC engines 312a-312n, 318a-318n can be configured to generate LDPC code or parity data. LDPC codes and parity data can be attached to the corresponding incoming data of the respective NVM dies to be written among the multiple NVM dies 316a-316n, 322a-322n. In one embodiment, the FM ASIC includes approximately 14 LDPC engines. In another embodiment, the FM ASIC includes fewer than approximately 54 LDPC engines.

[0041] LDPC codes and parity data can be used to locate and repair faulty bits during read and write operations on multiple NVM dies 316a-316n, 322a-322n. In one implementation, the high faulty bit count (FBC) corresponds to approximately 10.0% of the error correction code (ECC) or parity data size. For example, in solid-state drives (SSDs), programming to NVM (such as...) Figure 1 Approximately 10.0% of the data in the NVM 110 may be ECC or parity data. In another embodiment, a low FBC corresponds to approximately 33.3% of the ECC or parity data size. As the ECC or parity data size increases from approximately 10.0% to approximately 33.3%, the FBC decreases because the data includes a greater ability to find and repair faulty or incorrect bits. In another embodiment, each of the plurality of NVM dies 316a-316n, 322a-322n includes between approximately 10.0% and approximately 33.3% of the ECC or parity data associated with the corresponding stored data. Furthermore, each of the plurality of NVM dies 316a-316n, 322a-322n may have a bit error rate (BER) of approximately 0.2 or less. By including more ECC or parity data along with the corresponding data stored in NVM dies 316a-316n, 322a-322n, the BER can be reduced or improved, resulting in a BER value closer to approximately 0. The table below describes the power consumption and read performance improvements achieved by increasing the amount of ECC or parity data stored on each of the multiple NVM dies 316a-316n, 322a-322n.

[0042]

[0043] Table 1

[0044] The values ​​listed in Table 1 are not intended to be limiting, but rather to provide examples of possible implementations. While the total data storage capacity is lower when the ECC or parity data size is approximately 33.3% (i.e., low FBC) than when the ECC or parity data size is approximately 10.0% (i.e., high FBC), read performance increases from approximately 1.2 GB / s to approximately 4.8 GB / s, and power consumption decreases from approximately 0.200 watts (using approximately 10.0% parity size, or a high BER engine) to approximately 0.120 watts (using approximately 33.3% parity size, or a low BER engine). Therefore, data storage device 304 can have improved power consumption and read performance when the ECC or parity data size is larger.

[0045] The FE ASIC 306 includes multiple Reduced Instruction Set Computer (RISC) processing cores 308a-308n. In this description, for illustrative purposes, the RISC processing cores 308a-308n may be referred to as processing cores 308a-308n. Although a RISC processing core is described, other types of processing cores, such as CISC or other processor architectures, may be utilized in embodiments. For example, the FE ASIC 306 may include multiple processing cores with more than about five processing cores. In another embodiment, the number of processing cores is about 256 and about 512. Each of the multiple processing cores 308a-308n is configured to receive and execute database instructions from host 302. Database instructions may include one of select instructions, update instructions, and insert instructions. In addition to the previously mentioned instructions, database instructions may also include delete instructions. Furthermore, when receiving database instructions from host 302, FE ASIC 306 can allocate an appropriate number of processing cores from multiple processing cores 308a-308n to complete the requested database instructions.

[0046] Figure 4 The timing of database selection, updating, and insertion according to the disclosed implementation scheme is illustrated. Figure 4 The two databases shown are a full-schema database (such as MySQL) and a schema-less database (such as MongoDB). The following... Figures 5A to 5D The description discusses the differences between full-schema and schema-less databases. SELECT commands are used to retrieve one or more values ​​(i.e., data), such as rows or cells, from one or more tables in a database. UPDATE commands are used to update one or more values, such as rows or cells, from one or more tables in a database, where the UPDATE command updates the relevant data retrieved from the SELECT command. INSERT commands are used to insert one or more values ​​(such as rows or cells) into one or more tables in a database. It should be understood that while SELECT, UPDATE, and INSERT commands are illustrated, other database commands (such as DELETE commands) are relevant and can be associated with the disclosed implementation.

[0047] In the SELECT timing, MySQL database has a completion time of approximately 3.8 seconds, while MongoDB database has a completion time of approximately 34.47 seconds. In the UPDATE timing, MySQL database has a completion time of approximately 49.64 seconds, while MongoDB database has a completion time of approximately 0.9 seconds. In the INSERT timing, MySQL database has a completion time of approximately 1,914 seconds, while MongoDB database has a completion time of approximately 633 seconds. The times listed are not intended to be limiting, but rather to provide examples of possible implementations. Generally speaking, MongoDB database has a faster completion time than MySQL database. Furthermore, when queries (such as SELECT and UPDATE queries) are coupled together, MongoDB database has a faster completion time than MySQL database. MongoDB database may be better suited for large unstructured datasets, while MySQL database may be better suited for smaller datasets, as MySQL database may not be able to handle large volumes of unstructured data.

[0048] Figures 5A to 5D This is a graph illustrating the performance of various models based on the disclosed implementation scheme. It should be understood that while the results in the graphs are illustrated, other results can be achieved with different hardware, operating systems, and operating conditions. Database systems (such as...) Figure 3 The schema of the storage system 300 is relative to the structure of the stored data. For example, one or more NVM dies among multiple NVM dies 316a-316n, 322a-322n can be configured with a schema for storing data. Schema structures may include at least one of the following: indexes, encapsulations, procedures, functions, queues, triggers, types, sequences, materialized views, synonyms, database links, directories, Java schemas, and XML schemas. When the database (i.e., the schema) is modified (such as changing the data type of a column in the database), the schema needs to be updated with the relevant data types to store the updated data. The schema may store JavaScript Object Representation (JSON) documents, which is an open standard file format. For example, an open standard file format may be a data exchange format that uses text to store and transmit data objects consisting of attribute-value pairs and array data types.

[0049] There are two types of schemas: schema-based databases and schema-less databases. A schema-based database is a statically typed database where data is stored in specific columns based on data type. An example of a schema-based database is MySQL. However, a schema-less database is a dynamically typed database where the data type of columns can be editable or changeable. An example of a schema-less database is MongoDB. Schema-less database structures allow for more flexible storage of a wider variety of data types, such as documents, compared to schema-based database structures. In some implementations, a schema-less database can be considered not only SQL (NoSQL) databases, where NoSQL refers to databases that store documents, key values, wide columns, and graphs. Data stored in schema-based databases can be stored as text, such as in the Google Protocol Buffer (referred to as Protobuf in this paper) schema. Protobuf is a method for serializing structured data. The method involves an interface description language that describes the structure of some data and a program that generates text from the description representing the structured data.

[0050] Figure 5A This is a graph 500 showing the read times for various schema databases (such as Protobuf, Newtonsoft JSON, Newtonsoft Bson, ServiceStack JSON, and .NET XML). Protobuf read times are significantly faster than Newtonsoft JSON, Newtonsoft Bson, ServiceStack JSON, and .NET XML. For example, Protobuf has a read time of approximately 300 ms, while .NET XML has a read time of approximately 1050 ms. Compared to the next fastest read time of 1050 ms, Protobuf's read time is approximately 3.5 times faster.

[0051] Figure 5B This is a graph (525) showing the write time for various schema databases (such as Protobuf, Newtonsoft JSON, NewtonsoftBson, ServiceStack JSON, and .NET XML). For example, Protobuf has a write time of approximately 250 ms, and .NET XML has a read time of approximately 790 ms. Protobuf's read time is approximately 3.16 times faster than the next fastest read time of 790 ms.

[0052] Figure 5CThis is a graph 550 showing the combined read and write times for various schema databases (such as Protobuf, Newtonsoft JSON, Newtonsoft Bson, ServiceStack JSON, and .NET XML). For example, Protobuf has a combined read and write time of approximately 550 ms, and .NET XML has a combined read and write time of approximately 1840 ms. Protobuf's combined read and write time is approximately 3.35 times faster than the next fastest combined read and write time of 1840 ms.

[0053] Figure 5D This is a graph 575 showing the sizes of the resulting files for various schema databases (such as Protobuf, Newtonsoft JSON, Newtonsoft Bson, ServiceStack JSON, and .NET XML). For data writes to schema databases, the resulting file size is typically larger than the original file size. The resulting file size also includes metadata, parity data, etc. For example, Protobuf has a resulting file size of approximately 40,000,000 bytes, while Newtonsoft JSON, Newtonsoft Bson, and ServiceStack JSON each have a resulting file size of approximately 50,000,000 bytes. The Protobuf resulting file size is approximately 20% smaller than the next smallest resulting file size of 50,000,000 bytes.

[0054] Figure 6 This is a flowchart illustrating a method 600 for creating a schema from schema-less data according to the disclosed embodiment. The created schema can be a data serialization structure. At block 602, a controller (such as...) Figure 1 The controller 108) receives a portion of a file containing schema-less data. The file may include multiple records, each containing one or more fields corresponding to one or more data types. In one implementation, the file may include multiple field-defined units of document-based data. For example, MongoDB and other types of schema-less databases are document-based databases. Field-defined units of document-based data may include hierarchies, repeating fields, optional fields, and combinations of previously listed units.

[0055] At box 604, the controller may include a processor or processing core, such as processing cores 308a-308n, configured to identify or detect fields in a received document that includes schema-less data. Fields may include at least one of a field name, field type, repeating fields, and optional fields. Additionally, fields may include a field size and a required field. Furthermore, field types may be numeric, floating-point, boolean, string, enumerated, etc. The fields listed above are not intended to be limiting but rather to provide examples of possible implementations. In one implementation, each field may be specific to the data type of a field delimitation unit based on the document's data.

[0056] At box 606, the processor or processing core is configured to create a schema for schema-less data. The schema can be a statically typed schema with dynamic fields, allowing the fields of the statically typed schema to be updated or changed. Furthermore, due to... Figures 5A to 5D The other schemas shown offer performance advantages over the Protobuf schema, which can be formatted as Protobuf. The schemas listed previously are not intended to be restrictive, but rather to provide examples of possible implementations. Based on the created schema, each of the multiple records of schemaless data is transformed into a table.

[0057] At box 608, the pattern and data from the file are stored in the memory module, such as... Figure 3 One or more NVM dies from the 316a-316n and 322a-322n series. Each column of the table can correspond to a field of the schema, so that when storing schemas and data from a file, both the schema and the table are stored in the memory module.

[0058] Figure 7 This is a flowchart illustrating an automatic pattern detection and migration method 700 according to the disclosed implementation scheme. Figure 6 Method 600 can be similar in various aspects to method 700 described above. At block 702a, the controller (such as...) Figure 1 The controller 108) and / or processing cores (referred to herein as processors for illustrative purposes) (such as processing cores 308a-308n) are configured to generate new tables and associated schemas, where the number of columns and the data types of the columns have not yet been identified. The columns of the table may correspond to fields such as field name, field type, field size, and required fields, and additional attributes of the columns may include whether the field is optional and / or a repeating field. However, if an existing table is stored at a memory module, such as one or more NVM dies among multiple NVM dies 316a-316n, 322a-322n, then method 700 begins at block 702b and continues to block 710 of method 700.

[0059] At box 704, data from a first approximately 1MB text file is loaded, wherein the first approximately 1MB text file is a portion of a received data file that is schema-less or has a dynamically typed schema. While the first 1MB of the file is disclosed herein for at least initial processing, other portions of the file size may be utilized up to and including the entire file. Furthermore, although a text file is disclosed herein, those skilled in the art will understand that other file formats can be parsed according to the embodiments disclosed herein. In embodiments, the file may be in XML format, JSON format, or other formats used for storing data via a schema-less or dynamically typed schema database (such as MongoDB). An example can be seen in Figure 8. In some embodiments, unstructured and schema-less data may be used according to this disclosure, wherein data types, field names, etc., are programmatically determined, such as through lookup tables, algorithms, machine learning algorithms (e.g., classification and / or regression algorithms; via supervised or unsupervised learning methods), or other methods that can parse the data to determine its type and content in order to develop schemas for that data. The dimensions previously listed are not intended to be limiting, but rather to provide examples of possible embodiments.

[0060] At box 706, the controller and / or processing core is configured to recognize the fields and structure of a text file. In an implementation, when parsing a schema-less or dynamically typed schema-based database (such as MongoDB), the parsed fields include field names, field types, and a determination of whether a field is a repeating or optional field, and the schema structure includes a structure name, structure hierarchy, repeating structures, and optional structures. Furthermore, the text file may include multiple field delimited units of document-based data. At box 708, the controller and / or processor generates the schema structure based on the recognized fields and the structure of the text fields. In one implementation, the schema structure is a Protobuf structure, while other implementations may utilize different serialized data schemas. Furthermore, the generated schema structure is a data serialization structure.

[0061] At box 710, the controller and / or processor are configured to read data records from the received file and convert them into a recognizable structure of the schema created at box 708. After parsing the first portion of the data (e.g., the 1MB of text file data at box 704), additional data from the file may be consumed and parsed. At box 712, when the controller and / or processor detects a mismatch between additional data from the received file and the schema (such as a new field not present in the schema, a change in data type, or a missing field), the controller and / or processor sends the mismatched data to the exception queue of the exception handler. At the exception queue, at box 714, the controller and / or processor identifies the type of mismatch and updates the schema structure to remedy the mismatch. For example, the controller and / or processor may change or update a field type to match a mismatched data type and produce a new schema structure that reflects the update. Similarly, the controller and / or processor may add new fields to the schema, resulting in new columns in the table to allow missing fields to have a place in the data table, and may mark the new fields as required or optional. At box 716, the controller and / or processor transforms, appends, and reads all data records from the old schema structure into the updated schema structure, including any mismatched data. For example, it transforms previously transformed records of a data table into the updated schema.

[0062] After the process is completed at box 716, or if no mismatch has been identified, the controller and / or processor determines at box 718 whether the exception queue is empty. If the exception queue is not empty, at box 714, the controller and / or processor continues to identify mismatches and update the schema structure. However, if the exception queue is empty at box 718, the controller and / or processor determines at box 720 whether the last data record of the file has been reached. If the last data record of the file has not been reached, at box 710, the controller and / or processor continues to read the data record and convert it into the identified schema structure. Method 700 continues to box 712, and so on. When the last data record of the file has been reached at box 720, the schema detection and migration method 700 is completed at box 722. When method 700 is complete, the controller and / or processor can be configured to perform database operations on the schema's data table, such as queries, record insertions, record updates, and record deletions.

[0063] Figure 8A This is an example of document 800 based on the disclosed implementation scheme. Document 800 can be any of the previously mentioned documents, such as Figure 7The text file. File 800 is a JSON file containing a first record and a second record. The first record, indicated by the first opening brace in the leftmost column and the subsequent first closing brace in the same column, includes the following fields: _id, password, firstname, lastname, and contacts. The "contacts" field also includes an internal structure with an array of values ​​indicated by brackets [], which includes the following fields: method and value. Quotation marks "" indicate text values. For example, for the first id, _id = 1, the password is stored as the text value "7f1afdbe". If only the first record exists in the table of the database, each field in the first record is considered either optional or required, thus indicating that each field is required to contain a value.

[0064] However, because document 800 also includes a second record indicated by the second opening brace in the leftmost column and the subsequent second closing brace in the same column as the first opening brace, which includes the following fields: _id, password, firstname, and lastname. Since the second record does not include the "contacts" field, the "contacts" field is converted to an optional field.

[0065] Figure 8B This is an example of document 850 based on the disclosed implementation scheme. Document 850 can be any of the previously mentioned documents, such as Figure 7 The text file. File 850 is an XML file containing the first record. The first record includes the fields “customer_order number”, “date”, “lines”, “customer”, and “payment”. The “lines” field includes subfields such as “line no='1'” and “line no='2'”, where each subfield in the “lines” field includes the fields “item”, “quantity”, and “price”. The “customer” field includes subfields such as “name”, “street”, “city”, and “post_code”. Similarly, the “payment” field includes subfields such as “card_issuer”, “card_number”, and “expiration_date month”. Each of the fields, subfields, and sub-subfields has a data type associated with the entry. For example, the “customer_order number” field is identified by an integer with zero padding, and the “date” field is formatted as “YYYY-MM-DD”.

[0066] Furthermore, each field can be considered an enumeration type dataset. Duplicate values ​​in the enum dataset can be detected for the same field, such as "T / F", "M / F", "Work / Home / Cellular", etc. Controllers and / or processors can be configured to collect up to, but not limited to, approximately 100 historical values ​​for each field, where controllers and / or processors can detect values ​​or fields that do not exist in the enumeration list of values.

[0067] The performance of data storage devices can be improved by generating and actively modifying statically typed schema databases for schema-less data.

[0068] In one embodiment, a data storage device is disclosed, comprising: one or more memory modules; and a controller including a processor configured to perform a method for data pattern detection and migration. In this embodiment, the method includes: receiving a portion of a file comprising schema-less data; identifying one of field names, field types, repeating fields, and optional fields in the schema-less data; creating a schema for the schema-less data including the fields; and storing the schema and data from the file in the one or more memory modules.

[0069] The file includes multiple records. The method also includes converting each of the multiple records into a table based on the schema. Storing the schema and data from the file includes storing the schema and the table. The method also includes providing an exception handler with a mismatch field when a mismatch field is identified in a record. The mismatch field includes one of a new field, a changed field type, or a missing field. The schema is updated to an updated schema that has been updated based on the mismatch field. Previously converted records in the table are converted to the updated schema. Previously unconverted records in the file are added to the table based on the updated schema.

[0070] In another embodiment, a controller for a data storage device is disclosed, the controller including: I / O for one or more memory devices; and a processor configured to perform a method for data pattern detection and migration. In this embodiment, the method includes: receiving a portion of a file including a plurality of field delimiters of document-based data; detecting the data type of one of the plurality of field delimiters of the document-based data; defining a pattern of the one of the field delimiters of the document-based data based on the data type; and storing the file and the pattern.

[0071] The method further includes: defining a data table comprising multiple records based on the file and the schema; performing one of the following on the data table: querying, inserting records, updating records, and deleting records; and detecting field mismatches, including detecting one of the following: a new field not present in the schema, a change in data type, or a missing field. The method also includes generating a new schema by updating the schema based on the field mismatch. The updated schema includes one of the following: updating the schema to include the new field, updating the data type of the field, and updating the schema to a field specification that is either required or optional. The method also includes updating the data table based on the new schema. The method also includes converting additional data from the file into the data table based on the new schema. The controller also includes identifying one of the multiple field delimiting units of the document-based data as hierarchical, repetitive, or optional.

[0072] In another embodiment, a system for storing data is disclosed, the system comprising: one or more memory devices; and an SSD controller device configured to perform methods for data pattern detection and migration. In this embodiment, the method includes: detecting a field hierarchy in a portion of a document, the document comprising one of data including fields based on a structured or semi-structured document, data associated with the fields, and a data type associated with the data; defining a schema based on the field hierarchy, the fields, the data, and the data type; and defining a data table based on the schema and the document.

[0073] The method also includes identifying field mismatches in another part of the file based on the pattern. The method further includes generating a new pattern by modifying the pattern based on the field mismatch, and updating the table based on the new pattern. The method also includes performing one of the following operations on the data table: querying, inserting records, deleting records, and modifying records. The pattern is a data serialization structure.

[0074] While the foregoing describes embodiments of this disclosure, other and additional embodiments of this disclosure may be contemplated without departing from the basic scope of this disclosure, the scope of which is defined by the appended claims.

Claims

1. A data storage device, comprising: One or more memory modules; and The controller includes a processor configured to perform a method for data pattern detection and migration, the method comprising: Receive a portion of a file that includes schema-less data, wherein the schema-less data comprises multiple records; In the schema-less data, identify one or more fields of a first record of the plurality of records, wherein each of the one or more fields corresponds to a corresponding data type of one or more data types; Based on the identification, a schema with one or more columns is created for the schema-less data, wherein: Each of the one or more columns corresponds to a corresponding field in one or more fields of the first record of the plurality of records; Each of one or more fields in the first record is a required field; Each value in each entry of the required fields is required; Each of the one or more fields can be changed from a required field to an optional field; and Each value in each entry of the optional fields is optional; The mode and the modeless data from the file are stored in the one or more memory modules; and In response to the determination that a corresponding field is missing in one or more fields of the second record of the plurality of records of the schema-less data, the column of the stored schema is changed from the required field to the optional field, wherein the optional field exists in the stored schema.

2. The data storage device of claim 1, wherein the method further comprises converting each of the plurality of records into a table according to the pattern, and wherein storing the pattern and data from the file includes storing the pattern and the table.

3. The data storage device according to claim 2, wherein the method further includes providing the field mismatch to an exception handler when identifying a mismatch field in a record.

4. The data storage device according to claim 3, wherein the mismatched field includes one of a new field, a changed field type, and a missing field.

5. The data storage device of claim 3, wherein the mode is updated to an updated mode that has been updated based on the mismatch field.

6. The data storage device of claim 5, wherein the previously converted records of the table are converted into the updated schema.

7. The data storage device of claim 5, wherein previously unconverted records of the file are added to the table based on the updated schema.

8. A controller for a data storage device, comprising: I / O refers to input / output to one or more memory devices; and A processor configured to perform a method for data pattern detection and migration, the method comprising: Receive a portion of a file, the file comprising multiple field-defined units based on document data; Detect the data type of the first field delimiter among the plurality of field delimiters of document-based data; The pattern of the first field delimiting unit among the plurality of field delimiting units of document-based data is defined based on the data type, wherein: The schema includes one or more columns, wherein: Each of the one or more columns corresponds to the data type being detected; Each of the one or more columns in the first field delimiting unit is a required field; Each value for each entry in a column that has required entries is required; Each column can be changed from a required field to an optional field; and Each value in each entry of the optional fields is optional; Store the file and the schema; and In response to the determination that the corresponding data type of the second field delimiter of the plurality of field delimiters is missing, the column of the stored pattern is changed from the required field to the optional field, wherein the optional field exists in the stored pattern.

9. The controller of claim 8, wherein the method further comprises: A data table comprising multiple records is defined based on the file and the pattern.

10. The controller of claim 9, wherein the method further comprises performing one of querying, inserting, updating and deleting records on the data table.

11. The controller of claim 9, wherein the method further comprises detecting field mismatch, including detecting one of a new field not present in the pattern, a change in data type, or a missing field.

12. The controller of claim 11, wherein the method further comprises generating a new pattern by updating the pattern based on the field mismatch, wherein updating the pattern includes one of the following: Update the schema to include the new field; Update the data type of the field; as well as Update the schema to specify the field of either the required field or the optional field.

13. The controller of claim 12, wherein the method further comprises updating the data table based on the new pattern.

14. The controller of claim 12, wherein the method further comprises converting additional data from the file into the data table based on the new pattern.

15. The controller of claim 8, further comprising identifying one of the plurality of field delimiting units of the document-based data as hierarchical, repetitive, and optionally.

16. A system for storing data, comprising: One or more memory devices; and A solid-state drive controller, or SSD controller, is configured to perform a method for data pattern detection and migration, the method comprising: Detecting the field hierarchy in a portion of a document, the document comprising data based on a structured or semi-structured document, wherein the portion of the document comprises multiple fields, data associated with each of the multiple fields, and a data type associated with the data; The pattern is defined based on the aforementioned field hierarchy, wherein: The schema includes one or more columns, wherein: Each column corresponds to a specific field among the multiple fields; Each of the plurality of fields in the portion of the document is a required field; Each value in each entry of the required fields is required; Each of the plurality of fields can be changed from a required field to an optional field; and Each value in each entry of the optional fields is optional; The data table is defined based on the pattern and the file; In response to determining that a corresponding field is missing in the next part of the file, the column of the pattern is changed from the required field to the optional field, wherein the optional field exists in the stored pattern.

17. The system of claim 16, the method further comprising identifying field mismatches in another portion of the file based on the pattern.

18. The system of claim 17, wherein the method further comprises generating a new schema based on modifying the schema according to the field mismatch, and updating the table based on the new schema.

19. The system of claim 18, the method further comprising performing one of querying the data table, inserting a record, deleting a record, and modifying a record.

20. The system of claim 16, wherein the mode is a data serialization structure.

Images