Unstructured and structured data global namespace management method and device
By uniformly managing structured and unstructured data in the Hubble Store global namespace, the problems of low data processing efficiency, limited storage capacity and insufficient analysis capabilities in the existing technology are solved, and efficient data processing and storage are achieved, supporting Seamless data access and AI training acceleration in different environments.
Patent Information
- Application Number
- CN202410915672.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-09
- Publication Date
- 2024-11-15
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
When processing and storing large-scale structured and unstructured data, the existing technology has problems such as low data processing efficiency, limited storage capacity, insufficient data analysis capabilities, and inconsistent namespaces of different data types, resulting in complex data processing and Inefficient.
By uniformly managing structured and unstructured data in the global namespace provided by the Hubble Store, leveraging the global namespace for data access, moving, synchronizing and query analysis, and expanding storage space through the Hubble Store to accelerate AI training and data pipelines, Achieve improvement of data processing efficiency and expansion of storage capacity.
Improve data processing efficiency, expand storage capacity, optimize data synchronization mechanism, improve system scalability, enhance data analysis capabilities, support seamless movement and access data between local and cloud environments, significantly improving AI training and the efficiency of data pipelines.
Smart Images

Figure CN118964684A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of data storage and access technology, and in particular to a method and device for managing a global namespace of unstructured and structured data. Background Art
[0002] In the field of data-intensive computing, traditional data storage and processing methods face many challenges. With the rapid development of artificial intelligence and machine learning technologies, there is a growing demand for efficient processing and analysis of large-scale structured and unstructured data. Structured data, such as tables in databases, are easy to query and analyze, but they usually cannot accommodate large amounts of unstructured data, such as text, images, and videos. Unstructured data, although containing rich information, is more complex to process and analyze.
[0003] In recent years, some solutions have attempted to solve this problem through distributed storage systems, but they often require compromises between performance, capacity, transactionality, and analytics. For example, some systems may provide high-performance data access at the expense of storage capacity; others may support large-scale data analysis but cannot effectively process real-time data streams.
[0004] In the big data ecosystem, HDFS is used as a distributed file system to manage file metadata that can store unstructured data such as text. Hbase is a KV database based on HDFS storage, and Hive is a structured data warehouse based on HDFS. The namespace for unstructured data such as text is presented by HDFS, which provides a namespace based on the file directory structure. The namespace for KV data is presented by Hbase, and the namespace can be viewed through Hbase's SDK, Cli, etc. Hive's Metastroe service manages the namespace for structured data. The above three types of data are stored in the form of files on HDFS, but the namespaces of these three types of data are not unified and are presented in different components. Their use is also restricted because they are in different namespaces. The three data structures are not connected, and different interfaces need to be called for processing, and the conversion between data requires ETL.
[0005] Data solutions from cloud service providers: Cloud service providers such as AWS, Azure, and Google Cloud Platform provide a range of data storage and processing services, including object storage, file storage, and database services. These services are usually highly scalable and flexible, and can support data processing needs of all sizes.
[0006] Distributed file systems and object storage: Open source solutions such as Ceph and Hadoop HDFS enable the construction of highly scalable storage systems on commodity hardware. These systems are particularly suitable for application scenarios that need to process large amounts of unstructured data.
[0007] Specialized big data and analytics platforms: Solutions provided by companies such as Snowflake and Databricks, which focus on data warehousing and big data analytics and provide powerful data query and analysis capabilities.
[0008] Emerging AI computing cloud services: Services provided by companies such as Coreweave, which provide computing and storage resources optimized specifically for AI and machine learning workloads.
[0009] Each data solution has its own unique advantages and limitations. Which solution you choose depends on a variety of factors, including the type of data, data volume, performance requirements, cost considerations, and user familiarity with the technology.
[0010] Hubble Store combines storage, database, and computing functions through a unified software platform, aiming to provide the infrastructure required for AI and deep learning for modern data centers and cloud environments. The platform adopts a new distributed system architecture, eliminating the communication or interdependency between machines running system logic, thereby achieving the ability to parallel read and write operations at any scale. In addition, a global namespace is provided to support structured query analysis tools such as Presto, Trino, Spark, and AI training tools such as PyTorch and TensorFlow, effectively unifying the processing of structured and unstructured data.
[0011] Based on this technical background, the present invention studies a method and device for managing a global namespace of unstructured and structured data. Summary of the invention
[0012] In view of the deficiencies in the prior art, the present invention provides a method and device for managing a global namespace for unstructured and structured data. The method uniformly manages structured and unstructured data in different environments through a global namespace, thereby improving data processing efficiency, expanding storage capacity, optimizing data synchronization mechanism, and enhancing system scalability, effectively solving the problems of low data processing efficiency, limited storage capacity, and insufficient data analysis capabilities in the prior art.
[0013] In order to achieve the above object, a first aspect of the present invention provides a method for managing a global namespace of unstructured and structured data, comprising: Store unstructured and structured data in the global namespace provided by Hubble Store; Accessing, moving, synchronizing or querying and analyzing the unstructured and structured data through the global namespace; Hubble Store is used to expand storage space and accelerate AI training and data pipelines to meet the storage space requirements for AI training and the access speed requirements for unstructured and structured data.
[0014] A second aspect of the present invention provides a management device for unstructured and structured data global namespaces, comprising: The spatial storage module is used to store unstructured and structured data in the global namespace provided by Hubble Store; A data operation module, used for accessing, moving, synchronizing or querying and analyzing the unstructured and structured data through the global namespace; The acceleration module is used to expand the storage space using Hubble Store and accelerate AI training and data pipelines to meet the storage space requirements during AI training and the access speed requirements for the unstructured and structured data.
[0015] A third aspect of the present invention provides an electronic device, the electronic device comprising: A memory storing executable instructions; A processor, wherein the processor runs the executable instructions in the memory to implement the method for managing the global namespace of unstructured and structured data according to the first aspect.
[0016] A fourth aspect of the present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the method for managing the global namespace of unstructured and structured data described in the first aspect.
[0017] The beneficial effects of the present invention include: (1) The method for managing the global namespace of unstructured and structured data proposed in the present invention uniformly manages structured and unstructured data in different environments through the global namespace, thereby improving data processing efficiency, expanding storage capacity, optimizing the data synchronization mechanism, and enhancing system scalability, effectively solving the problems of low data processing efficiency, limited storage capacity, and insufficient data analysis capabilities in the prior art.
[0018] (2) The management method of the global namespace of unstructured and structured data proposed in the present invention improves data processing efficiency, expands storage capacity, and enhances data analysis capabilities, providing an ideal choice for enterprises that process large-scale structured and unstructured data in modern data centers and cloud environments.
[0019] (3) The present invention proposes a method for managing the global namespace of unstructured and structured data. Through the global namespace, users can seamlessly move and access data between local and cloud environments without having to manage multiple data copies. At the same time, the Hubble Store accelerates AI and data pipelines, making feature engineering more efficient.
[0020] Other features and advantages of the present invention will be described in detail in the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS
[0021] The above and other objects, features and advantages of the present invention will become more apparent through a more detailed description of exemplary embodiments of the present invention with reference to the accompanying drawings.
[0022] Figure 1 A flow chart of the method for managing the global namespace of unstructured and structured data proposed by the present invention. DETAILED DESCRIPTION
[0023] The preferred embodiments of the present invention will be described in more detail below. Although the preferred embodiments of the present invention are described below, it should be understood that the present invention can be implemented in various forms and should not be limited to the embodiments set forth herein.
[0024] The present invention provides a method for managing the global namespace of unstructured and structured data, such as Figure 1 As shown, including: Store unstructured and structured data in the global namespace provided by Hubble Store; Access, move, synchronize, or query and analyze unstructured and structured data through a global namespace; Use Hubble Store to expand storage space and accelerate AI training and data pipelines to meet the storage space requirements for AI training and the access speed requirements for unstructured and structured data.
[0025] In the present invention, structured and unstructured data are uniformly managed in different environments through a global namespace, which improves data processing efficiency, expands storage capacity, optimizes data synchronization mechanism, and enhances system scalability, effectively solving the problems of low data processing efficiency, limited storage capacity, and insufficient data analysis capabilities in the prior art.
[0026] According to the present invention, storing unstructured and structured data in the global namespace provided by Hubble Store includes: Use Hubble Store to abstract data from a protocol. In the global namespace, abstracted data is stored as unstructured and structured data.
[0027] According to the present invention, accessing unstructured and structured data through a global namespace includes: Access unstructured and structured data from a global namespace using the same or different protocols as one; All protocols have equal access to unstructured and structured data.
[0028] Preferably, accessing and moving unstructured and structured data through a global namespace includes: Through the global namespace, users can move and access unstructured and structured data between local and cloud environments.
[0029] According to the present invention, synchronizing unstructured and structured data through a global namespace includes: Synchronize unstructured and structured data across different compute, analysis, and processing environments through a global namespace.
[0030] In the present invention, through the global namespace, users can seamlessly move and access data between local and cloud environments without having to manage multiple data copies. At the same time, the Hubble Store accelerates AI and data pipelines, making feature engineering more efficient.
[0031] According to the present invention, query analysis tools include Presto, Trino and Spark; Tools for AI training include PyTorch and TensorFlow.
[0032] Preferably, using Hubble Store to accelerate AI training and data pipelines includes: Accelerate AI training and data pipelines through Hubble Store integration with Apache Spark.
[0033] In the present invention, data processing efficiency is improved, storage capacity is expanded, and data analysis capabilities are enhanced, providing an ideal choice for enterprises that process large-scale structured and unstructured data in modern data centers and cloud environments.
[0034] The method of the present invention solves the following technical problems: 1) Improve data processing efficiency: By introducing a new data processing architecture, the present invention can provide faster data access and processing speed without sacrificing performance, thereby significantly improving the overall data processing efficiency; 2) Expand storage capacity: Using advanced data compression and storage technologies, more data can be stored in limited physical space while maintaining data integrity and accessibility; 3) Enhanced data analysis capabilities: It provides a unified platform that supports multiple data analysis tools, enabling users to conduct in-depth analysis of structured and unstructured data to gain more valuable insights and knowledge; 3) Optimize data synchronization mechanism: enable data to be quickly and consistently synchronized between different computing environments, supporting more complex data analysis tasks and real-time data stream processing; 4) Improve system scalability: The data processing platform is designed to be highly scalable and can adapt to the growing amount of data and increasingly complex data processing needs.
[0035] 5) Separation of data processing and persistence: The Hubble Store architecture separates data processing logic from data persistence, allowing the system to independently scale capacity and performance. This architecture allows for improved scalability, flexibility, and performance under different workloads and requirements.
[0036] The present invention will be described in more detail below by way of examples. Example
[0037] This embodiment proposes a method for managing a global namespace of unstructured and structured data. In terms of data management, by designing Hubble Store to abstract data from protocols, applications are allowed to write data using one protocol and read data through any other protocol, while retaining permissions and providing equal performance access across each protocol. As an integrated unstructured and structured namespace, this universal data platform makes it possible to simply catalog files and objects at any scale. Hubble Store considers the data storage mode from the perspective of AI native. Considering that AI needs both unstructured data such as text, images, and videos as well as structured data, in order to smooth out the problem of different namespaces between different data, Hubble Store upgrades the file system namespace to make the file system namespace fully compatible with structured and unstructured data. The specific implementation pseudo code is as follows: SELECT parent_path,name,size,uid FROM "hubble_catalog_table: / " WHERE uid = 5800 AND size>1024 AND extension = 'pdf'; SELECT sum(used) FROM "hubble_catalog_table: / storetest" WHERE uid = 5800 AND size>1024; From the above two SQL statements, we can see that the table name comes after From. The table of a traditional relational database is generally named something like "database1.table1". Hubble store uses file namespace to map table names, which directly unifies unstructured files and structured data from the file namespace layer. Unstructured files can be accessed through file namespaces, and structured files can also be accessed through file namespaces. They are integrated. Data files can be directly operated based on object storage and file system protocols, and data can also be accessed through structured queries. Hubble store adds a lock mechanism to file layer operations to prevent data conflicts when different protocols access the same file. In terms of AI training deep learning data platform, Hubble Store bridges the gap between deep learning and traditional data platforms, providing customers with a simple experience while meeting the needs of deep learning applications for data types, data scale, and data locality. The platform points to the relatively basic form of generative artificial intelligence used by big languages today by building an architecture that can store and organize EB-level data and schedule computing functions on a group of distributed artificial intelligence supercomputers. Through these examples, Hubble Store demonstrates its ability to process large-scale structured and unstructured data in modern data centers and cloud environments, as well as its support for deep learning tasks. Example
[0038] This embodiment provides a method for managing the global namespace of unstructured and structured data, such as Figure 1 As shown, including: Store unstructured and structured data in the global namespace provided by Hubble Store; Access, move, synchronize, or query and analyze unstructured and structured data through a global namespace; Use Hubble Store to expand storage space and accelerate AI training and data pipelines to meet the storage space requirements for AI training and the access speed requirements for unstructured and structured data; In this embodiment, storing unstructured and structured data in the global namespace provided by Hubble Store includes: Use Hubble Store to abstract data from a protocol. In the global namespace, abstracted data is stored as unstructured and structured data; Access to unstructured and structured data through the global namespace includes: Access unstructured and structured data from a global namespace using the same or different protocols as one; All protocols have equal access to unstructured and structured data; Access and move unstructured and structured data through the global namespace, including: Through the global namespace, users can move and access unstructured and structured data between local and cloud environments; In this embodiment, synchronizing unstructured and structured data through the global namespace includes: Synchronize unstructured and structured data across different computing, analysis, and processing environments through a global namespace; In this embodiment, query analysis tools include Presto, Trino, and Spark; Tools for AI training include PyTorch and TensorFlow; Using Hubble Store to accelerate AI training and data pipelines includes: Accelerate AI training and data pipelines through Hubble Store integration with Apache Spark. Example
[0039] This embodiment provides a management device for a global namespace of unstructured and structured data, including: The spatial storage module is used to store unstructured and structured data in the global namespace provided by Hubble Store; Data operation module, used to access, move, synchronize or query and analyze unstructured and structured data through the global namespace; The acceleration module is used to expand storage space using Hubble Store and accelerate AI training and data pipelines to meet the storage space requirements during AI training and the access speed requirements for unstructured and structured data. In this embodiment, storing unstructured and structured data in the global namespace provided by Hubble Store includes: Use Hubble Store to abstract data from a protocol. In the global namespace, abstracted data is stored as unstructured and structured data; Access to unstructured and structured data through the global namespace includes: Access unstructured and structured data from a global namespace using the same or different protocols as one; All protocols have equal access to unstructured and structured data; Access and move unstructured and structured data through the global namespace, including: Through the global namespace, users can move and access unstructured and structured data between local and cloud environments; In this embodiment, synchronizing unstructured and structured data through the global namespace includes: Synchronize unstructured and structured data across different computing, analysis, and processing environments through a global namespace; In this embodiment, query analysis tools include Presto, Trino, and Spark; Tools for AI training include PyTorch and TensorFlow; Using Hubble Store to accelerate AI training and data pipelines includes: Accelerate AI training and data pipelines through Hubble Store integration with Apache Spark. Example
[0040] An embodiment of the present invention provides an electronic device including a memory and a processor. A memory storing executable instructions; The processor runs the executable instructions in the memory to implement the management method of the global namespace of unstructured and structured data.
[0041] The memory is used to store non-temporary computer-readable instructions. Specifically, the memory may include one or more computer program products, and the computer program product may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, a random access memory (RAM) and / or a cache memory (cache), etc. The non-volatile memory may include, for example, a read-only memory (ROM), a hard disk, a flash memory, etc.
[0042] The processor may be a central processing unit (CPU) or other forms of processing units with data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device to perform desired functions. In one embodiment of the present invention, the processor is used to execute the computer-readable instructions stored in the memory.
[0043] Those skilled in the art should understand that in order to solve the technical problem of how to obtain a good user experience, the present embodiment may also include well-known structures such as a communication bus and an interface, and these well-known structures should also be included in the protection scope of the present invention.
[0044] For detailed description of this embodiment, reference may be made to the corresponding descriptions in the aforementioned embodiments, which will not be repeated here. Example
[0045] An embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, a method for managing a global namespace of unstructured and structured data is implemented.
[0046] The computer-readable storage medium according to the embodiment of the present invention stores non-transitory computer-readable instructions, and when the non-transitory computer-readable instructions are executed by a processor, all or part of the steps of the above-mentioned methods of the embodiments of the present invention are executed.
[0047] The above-mentioned computer-readable storage media include, but are not limited to: optical storage media (e.g., CD-ROM and DVD), magneto-optical storage media (e.g., MO), magnetic storage media (e.g., magnetic tape or mobile hard disk), media with built-in rewritable non-volatile memory (e.g., memory card) and media with built-in ROM (e.g., ROM box).
[0048] The embodiment of the present invention proposes a method for managing a global namespace for unstructured and structured data, which uniformly manages structured and unstructured data in different environments through a global namespace, thereby improving data processing efficiency, expanding storage capacity, optimizing data synchronization mechanism, and enhancing system scalability, and effectively solving the problems of low data processing efficiency, limited storage capacity, and insufficient data analysis capabilities in the prior art.
[0049] The embodiments of the present invention have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments.
Claims
1. A method for managing a global namespace of unstructured and structured data, characterized in that: include: Store unstructured and structured data in the global namespace provided by Hubble Store; Accessing, moving, synchronizing or querying and analyzing the unstructured and structured data through the global namespace; Hubble Store is used to expand storage space and accelerate AI training and data pipelines to meet the storage space requirements for AI training and the access speed requirements for unstructured and structured data.
2. The method according to claim 1, characterized in that Storing unstructured and structured data in the global namespace provided by Hubble Store includes: Use Hubble Store to abstract data from a protocol. In the global namespace, the abstracted data is stored as the unstructured and structured data.
3. The method according to claim 2, characterized in that Accessing the unstructured and structured data through the global namespace includes: Accessing the unstructured and structured data from the global namespace using a protocol that is the same as or different from the certain protocol; All protocols have equal access to the unstructured and structured data.
4. The method according to claim 1, characterized in that Accessing and moving the unstructured and structured data through the global namespace includes: Through the global namespace, users can move and access the unstructured and structured data between local and cloud environments.
5. The method according to claim 1, characterized in that Synchronizing the unstructured and structured data through the global namespace includes: The unstructured and structured data are synchronized between different computing, analysis and processing environments through the global namespace.
6. The method according to claim 1, characterized in that The query analysis tools include Presto, Trino and Spark; The AI training tools include PyTorch and TensorFlow.
7. The method according to claim 6, characterized in that Using Hubble Store to accelerate AI training and data pipelines includes: Accelerate AI training and data pipelines through Hubble Store integration with Apache Spark.
8. A device for managing global namespaces of unstructured and structured data, characterized in that: include: The spatial storage module is used to store unstructured and structured data in the global namespace provided by Hubble Store; A data operation module, used for accessing, moving, synchronizing or querying and analyzing the unstructured and structured data through the global namespace; The acceleration module is used to expand the storage space using Hubble Store and accelerate AI training and data pipelines to meet the storage space requirements during AI training and the access speed requirements for the unstructured and structured data.
9. An electronic device, characterized in that: The electronic device comprises: A memory storing executable instructions; A processor, wherein the processor runs the executable instructions in the memory to implement the method for managing the global namespace of unstructured and structured data according to any one of claims 1-7.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method for managing the global namespace of unstructured and structured data described in any one of claims 1 to 7 is implemented.
Citation Information
Patent Citations
Model training method and system, data processing method and system and electronic equipment
CN116070113A
Data query method and device and storage medium
CN117076497A
Supporting unified querying over autonomous unstructured and structured databases
US20090248619A1
