A system for partitioning and storing data in artificial intelligence systems

Abstract

This invention discloses a system for partitioning and storing data from an artificial intelligence system, relating to the field of data storage technology; it includes: Step 1: establishing a system for partitioning and storing data from an artificial intelligence system; Step 2: identifying various types of data generated by the artificial intelligence system through a data classification module, classifying the data according to data type, data characteristics, and access patterns; Step 3: formulating a dynamic partitioning strategy based on the data classification results and system load through a partitioning strategy formulation module; Step 4: routing received data requests to the correct data table through a data routing module; Step 5: receiving call requests through a data retrieval module, parsing the data characteristics in the request, quickly locating the corresponding partition based on the data characteristics, extracting the required data from the partition, and returning the data to the artificial intelligence system after format conversion; Step 6: managing physical storage resources through a storage management module; Step 7: maintaining the metadata information of the partitioned storage system through a metadata management module.

Description

Technical Field

[0001] This invention discloses a system for partitioning and storing data in artificial intelligence systems, relating to the field of data storage technology. Background Technology

[0002] With the rapid development of artificial intelligence technology, the amount of data generated by various AI systems is growing exponentially. Existing data storage methods suffer from the following problems when dealing with large-scale AI data:

[0003] Single-table storage capacity limitation: When the amount of data reaches the TB or even PB level, single-table storage will cause a sharp drop in query efficiency.

[0004] Data access performance bottleneck: A large number of concurrent accesses to the same data table will cause system response delays.

[0005] Data management complexity: Mixed storage of different types of AI data, such as training data, inference data, and model parameters, increases the difficulty of management.

[0006] Insufficient scalability: Existing storage architectures struggle to adapt to the dynamically changing data requirements of AI systems.

[0007] While there are some database sharding and table partitioning solutions in the existing technology, these solutions are mainly designed for existing business systems and cannot effectively handle the unique data characteristics of AI systems, such as a large amount of unstructured data, complex data relationships, and high real-time requirements. Summary of the Invention

[0008] This invention addresses the problems of existing technologies by providing a system for partitioning and storing data in artificial intelligence systems. It aims to solve the problems of massive data storage and efficient querying in artificial intelligence applications by improving data storage efficiency and query performance through a reasonable data partitioning strategy.

[0009] The specific solution proposed in this invention is as follows:

[0010] This invention provides a method for partitioning and storing data in an artificial intelligence system, comprising:

[0011] Step 1: Establish a system for partitioning and storing data from the artificial intelligence system. This system includes a data classification module, a partitioning strategy formulation module, a data routing module, a data retrieval module, a storage management module, and a metadata management module.

[0012] Step 2: The data classification module identifies various types of data generated by the artificial intelligence system and classifies the data according to data type, data characteristics, and access patterns.

[0013] Step 3: Based on the data classification results and system load, the table partitioning strategy formulation module formulates a dynamic table partitioning strategy, including: time-based partitioning, storing data according to its generation time; data-feature-based partitioning, storing data according to its semantic features; access-frequency partitioning, separating hot and cold data based on access frequency; partitioning specific fields according to business needs; and partitioning by data source, storing data from different sources in their respective partitions.

[0014] Step 4: The data routing module routes the received data requests to the correct data tables, including: write routing, which determines the target table for writing data based on the table partitioning strategy; query routing, which locates the data table to be accessed based on the query conditions; and update routing, which forwards the routes for processing data update requests.

[0015] Step 5: Receive the call request through the data call module, parse the data characteristics in the request, quickly locate the corresponding partition table based on the data characteristics, extract the required data from the partition table, and return the data to the artificial intelligence system after format conversion;

[0016] Step 6: Manage physical storage resources through the storage management module.

[0017] Step 7: Maintain the metadata information of the partitioned storage system through the metadata management module.

[0018] Furthermore, in step 2 of the method, the data classification module classifies the data according to data type, data characteristics, and access patterns, including: parsing the format and structure of the data to distinguish between structured data, semi-structured data, and unstructured data; dividing the data into different time periods by hour, day, or month according to the time of data generation; classifying the data into levels according to the importance of the data by pre-setting evaluation criteria; and distinguishing data from different users and different functional modules according to the data source.

[0019] Furthermore, in step 3 of the method, the table partitioning strategy formulation module generates a specific table partitioning scheme based on the data classification results and system load. For structured training data, the data is partitioned according to the data generation time, and time-dimensional partitioning is formulated. The training data generated each day is stored in an independent partition, or the data is partitioned according to specific fields according to business needs. For semi-structured user interaction data, the data is partitioned according to the data source, and the interaction data of different user groups are stored in the corresponding partitions.

[0020] Furthermore, in step 4 of the method, when the received data request is routed to the correct data table through the data routing module, a data verification mechanism is adopted to perform integrity verification on the written data. If data is missing or incorrect, it is promptly retransmitted or corrected. At the same time, the storage status of the partitioned table is monitored in real time. When the storage space of the partitioned table reaches the warning value, a prompt message is issued and the storage space is automatically expanded.

[0021] Furthermore, in step 2 of the method, the data classification module pre-sets evaluation criteria based on the importance of the data and divides the data into three levels: high, medium, and low.

[0022] Furthermore, in step 3 of the method, a table partitioning strategy formulation module is used to set up redundant backup tables for high-importance data.

[0023] Furthermore, step 6 of the method specifically includes:

[0024] Perform index management: Establish appropriate index structures for each partitioned table;

[0025] Perform cache management: Maintain a memory cache for frequently accessed data.

[0026] Furthermore, in step 7 of the method, the metadata information of the sharded storage system is maintained through the metadata management module, including maintaining the table structure, sharding rules and data distribution information.

[0027] This invention also provides a system for table-based storage of data from artificial intelligence systems, including a data classification module, a table-based strategy formulation module, a data routing module, a data retrieval module, a storage management module, and a metadata management module.

[0028] The data classification module identifies various types of data generated by the artificial intelligence system and classifies the data according to data type, data characteristics, and access patterns.

[0029] The table partitioning strategy formulation module formulates dynamic table partitioning strategies based on data classification results and system load, including: time-based table partitioning, storing data according to its generation time; data feature-based table partitioning, classifying and storing data according to its semantic features; access frequency-based table partitioning, separating hot and cold data based on access frequency; table partitioning based on specified fields according to business needs; and table partitioning based on data source, storing data from different sources in their respective partitions.

[0030] The data routing module routes received data requests to the correct data tables, including: write routing, which determines the target table for data writing based on the table partitioning strategy; query routing, which locates the data table to be accessed based on the query conditions; and update routing, which forwards the routes for processing data update requests.

[0031] The data retrieval module receives retrieval requests, parses the data features in the requests, quickly locates the corresponding partitioned table based on the data features, extracts the required data from the partitioned table, and returns the data to the artificial intelligence system after format conversion.

[0032] The storage management module manages physical storage resources.

[0033] The metadata management module maintains the metadata information of the partitioned storage system.

[0034] The advantages of this invention are:

[0035] Improved storage efficiency: By employing a reasonable table partitioning strategy, performance issues caused by excessively large data in a single table are avoided;

[0036] Query performance has been optimized: intelligent routing based on query patterns reduces unnecessary data scanning;

[0037] Enhanced system scalability: Supports dynamic addition of table partitions and storage nodes to adapt to data volume growth;

[0038] Reduced management complexity: Automated table partitioning management reduces the need for manual intervention;

[0039] Improved system reliability: Distributed storage reduces the risk of single points of failure. Attached Figure Description

[0040] Figure 1 This is a schematic diagram of the execution flow of the method of the present invention. Detailed Implementation

[0041] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, so that those skilled in the art can better understand and implement the present invention. However, the embodiments described are not intended to limit the present invention.

[0042] Example 1

[0043] This invention provides a method for partitioning and storing data in an artificial intelligence system, comprising:

[0044] Step 1: Establish a system for storing data from artificial intelligence systems in sharded tables. The system includes a data classification module, a sharding strategy formulation module, a data routing module, a data retrieval module, a storage management module, and a metadata management module.

[0045] Step 2: Identify the various types of data generated by the artificial intelligence system through the data classification module, and classify the data according to data type, data characteristics, and access patterns.

[0046] The data classification module categorizes data based on data type, characteristics, and access patterns. This includes: parsing the data format and structure to distinguish between structured data (e.g., database table data), semi-structured data (e.g., JSON data), and unstructured data (e.g., images, audio data); dividing the data into time periods by hour, day, or month based on generation time; classifying data into levels based on pre-set evaluation criteria according to importance; and differentiating data from different users and functional modules based on source. Specifically, the data classification module allows for pre-setting evaluation criteria to classify data into high, medium, and low levels based on importance.

[0047] Step 3: Based on the data classification results and system load, the table partitioning strategy formulation module formulates a dynamic table partitioning strategy, including: time-based partitioning, storing data according to its generation time; data-feature-based partitioning, classifying and storing data according to its semantic features; access-frequency partitioning, separating hot and cold data based on access frequency; partitioning specific fields according to business needs; and partitioning by data source, storing data from different sources in their respective partitions.

[0048] The table partitioning strategy module can further generate specific table partitioning schemes based on data classification results and system load. For structured training data, table partitioning can be performed by data generation time, with training data generated each day stored in a separate table. Alternatively, table partitioning can be performed by specific fields according to business needs, such as serial numbers in industrial production, where one serial number is assigned for each production run, and the data is stored in the corresponding serial number. For semi-structured user interaction data, table partitioning can be performed by data source, with interaction data from different user groups stored in their respective tables.

[0049] The partitioning strategy module also allows you to set up redundant backup partitions for highly important data.

[0050] Step 4: The data routing module routes the received data requests to the correct data tables, including: write routing, which determines the target table for writing data based on the table partitioning strategy; query routing, which locates the data table to be accessed based on the query conditions; and update routing, which forwards the routes for processing data update requests.

[0051] When the data routing module routes the received data request to the correct data table, it employs a data verification mechanism to check the integrity of the written data. If data is missing or incorrect, it promptly retransmits or corrects it. Simultaneously, it monitors the storage status of the partitioned tables in real time. When the storage space of a partitioned table reaches a warning value, it issues a prompt message and automatically expands the storage space.

[0052] Step 5: Receive the call request through the data call module, parse the data characteristics in the request, quickly locate the corresponding partition table based on the data characteristics, extract the required data from the partition table, and return the data to the artificial intelligence system after format conversion.

[0053] Step 6: Manage physical storage resources through the storage management module, specifically including:

[0054] Perform index management: Establish appropriate index structures for each partitioned table;

[0055] Implement cache management: Maintain a memory cache for frequently accessed data. For example, during data retrieval, use a caching mechanism to temporarily store frequently used data in the cache, thereby improving the speed of data retrieval.

[0056] Step 7: Maintain the metadata information of the partitioned storage system through the metadata management module, which may include maintaining table structure, partitioning rules and data distribution.

[0057] Example 2

[0058] This invention also provides a system for table-based storage of data from artificial intelligence systems, including a data classification module, a table-based strategy formulation module, a data routing module, a data retrieval module, a storage management module, and a metadata management module.

[0059] The data classification module identifies various types of data generated by the artificial intelligence system and classifies the data according to data type, data characteristics, and access patterns.

[0060] The table partitioning strategy formulation module formulates dynamic table partitioning strategies based on data classification results and system load, including: time-based table partitioning, storing data according to its generation time; data feature-based table partitioning, classifying and storing data according to its semantic features; access frequency-based table partitioning, separating hot and cold data based on access frequency; table partitioning based on specified fields according to business needs; and table partitioning based on data source, storing data from different sources in their respective partitions.

[0061] The data routing module routes received data requests to the correct data tables, including: write routing, which determines the target table for data writing based on the table partitioning strategy; query routing, which locates the data table to be accessed based on the query conditions; and update routing, which forwards the routes for processing data update requests.

[0062] The data retrieval module receives retrieval requests, parses the data features in the requests, quickly locates the corresponding partitioned table based on the data features, extracts the required data from the partitioned table, and returns the data to the artificial intelligence system after format conversion.

[0063] The storage management module manages physical storage resources.

[0064] The metadata management module maintains the metadata information of the partitioned storage system.

[0065] The information interaction and execution process between the modules in the above system are based on the same concept as the method embodiment of the present invention, and the specific details can be found in the description in the method embodiment of the present invention, and will not be repeated here.

[0066] Similarly, the advantages of the system of the present invention are:

[0067] Improved storage efficiency: By employing a reasonable table partitioning strategy, performance issues caused by excessively large data in a single table are avoided;

[0068] Query performance has been optimized: intelligent routing based on query patterns reduces unnecessary data scanning;

[0069] Enhanced system scalability: Supports dynamic addition of table partitions and storage nodes to adapt to data volume growth;

[0070] Reduced management complexity: Automated table partitioning management reduces the need for manual intervention;

[0071] Improved system reliability: Distributed storage reduces the risk of single points of failure.

[0072] It should be noted that not all steps and modules in the above processes and system structures are mandatory; some steps or modules can be omitted as needed. The execution order of the steps is not fixed and can be adjusted as required. The system structures described in the above embodiments can be physical or logical structures. That is, some modules may be implemented by the same physical entity, or some modules may be implemented by multiple physical entities, or they may be implemented by certain components in multiple independent devices.

[0073] The above-described embodiments are merely preferred embodiments provided to fully illustrate the present invention, and the scope of protection of the present invention is not limited thereto. Equivalent substitutions or modifications made by those skilled in the art based on the present invention are all within the scope of protection of the present invention. The scope of protection of the present invention is defined by the claims.

Claims

1. A method for partitioning and storing data in an artificial intelligence system, characterized by: include: Step 1: Establish a system for partitioning and storing data from the artificial intelligence system. This system includes a data classification module, a partitioning strategy formulation module, a data routing module, a data retrieval module, a storage management module, and a metadata management module. Step 2: The data classification module identifies various types of data generated by the artificial intelligence system and classifies the data according to data type, data characteristics, and access patterns. Step 3: Based on the data classification results and system load, the table partitioning strategy formulation module formulates a dynamic table partitioning strategy, including: time-based partitioning, storing data according to its generation time; data-feature-based partitioning, storing data according to its semantic features; access-frequency partitioning, separating hot and cold data based on access frequency; partitioning specific fields according to business needs; and partitioning by data source, storing data from different sources in their respective partitions. Step 4: The data routing module routes the received data requests to the correct data tables, including: write routing, which determines the target table for writing data based on the table partitioning strategy; query routing, which locates the data table to be accessed based on the query conditions; and update routing, which forwards the routes for processing data update requests. Step 5: Receive the call request through the data call module, parse the data features in the request, quickly locate the corresponding partition table based on the data features, extract the required data from the partition table, and return the data to the artificial intelligence system after format conversion; Step 6: Manage physical storage resources through the storage management module. Step 7: Maintain the metadata information of the partitioned storage system through the metadata management module.

2. The method of claim 1, wherein In step 2, the data classification module classifies the data according to data type, data characteristics, and access patterns. This includes: parsing the data format and structure to distinguish between structured, semi-structured, and unstructured data; dividing the data into different time periods by hour, day, or month according to the time of data generation; classifying the data into levels based on pre-set evaluation criteria according to the importance of the data; and distinguishing data from different users and different functional modules based on the data source.

3. The method according to claim 1, characterized in that in step 3, the table partitioning strategy formulation module generates a specific table partitioning scheme based on the data classification results and system load, wherein for structured training data, the data is partitioned according to the data generation time, and time-dimensional partitioning is formulated, with the training data generated each day stored in an independent partitioning table, or partitioning is performed on specified fields according to business needs; for semi-structured user interaction data, the data is partitioned according to the data source, and the interaction data of different user groups are stored in the corresponding partitioning tables.

4. The method according to claim 3, characterized in that when the received data request is routed to the correct data table through the data routing module in step 4, a data verification mechanism is adopted to verify the integrity of the written data. If data is found to be missing or incorrect, it is promptly retransmitted or corrected. At the same time, the storage status of the partitioned table is monitored in real time. When the storage space of the partitioned table reaches the warning value, a prompt message is issued and the storage space is automatically expanded.

5. The method of claim 2, wherein In step 2, the data classification module pre-sets evaluation criteria based on the importance of the data and divides the data into three levels: high, medium, and low.

6. The method of claim 5, wherein In step 3, the table partitioning strategy module is used to set up redundant backup tables for high-importance data.

7. The method of claim 1 wherein Step 6 specifically includes: Perform index management: Establish appropriate index structures for each partitioned table; Perform cache management: Maintain a memory cache for frequently accessed data.

8. The method according to claim 1, characterized in that: In step 7, the metadata information of the partitioned storage system is maintained through the metadata management module, including maintaining the table structure, partitioning rules and data distribution information.

9. A system for partitioning and storing data in artificial intelligence systems, characterized in that: It includes modules for data classification, table partitioning strategy formulation, data routing, data retrieval, storage management, and metadata management. The data classification module identifies various types of data generated by the artificial intelligence system and classifies the data according to data type, data characteristics, and access patterns. The table partitioning strategy formulation module formulates dynamic table partitioning strategies based on data classification results and system load, including: time-based table partitioning, storing data according to its generation time; data feature-based table partitioning, classifying and storing data according to its semantic features; access frequency-based table partitioning, separating hot and cold data based on access frequency; table partitioning based on specified fields according to business needs; and table partitioning based on data source, storing data from different sources in their respective partitions. The data routing module routes received data requests to the correct data tables, including: write routing, which determines the target table for data writing based on the table partitioning strategy; query routing, which locates the data table to be accessed based on the query conditions; and update routing, which forwards the routes for processing data update requests. The data retrieval module receives retrieval requests, parses the data features in the requests, quickly locates the corresponding partitioned table based on the data features, extracts the required data from the partitioned table, and returns the data to the artificial intelligence system after format conversion. The storage management module manages physical storage resources. The metadata management module maintains the metadata information of the partitioned storage system.

Images