Method and device for constructing materialized view of computing power data of intelligent computing center

Abstract

The invention provides a method and a device for constructing a materialized view of computing power data of an intelligent computing center. The method comprises the following steps of: S1, acquiring a data file imported into a multimode distributed file system; s2, acquiring feature data of the data file, and storing the feature data into a database table of a multimode vector database of the multimode distributed file system; s3, acquiring an input target database table identifier, acquiring the feature data from the target database table by adopting the multimode distributed file system, performing preset metadata processing on the feature data, generating metadata, and generating a data materialized view based on the metadata; and S4, acquiring a data directory corresponding to the data materialization view output by the multimode distributed file system. Therefore, the time occupied by data processing can be reduced, the labor cost is saved, the data processing efficiency of the intelligent computing center is improved, and the computing power of the intelligent computing center is improved.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of computing power technology, and in particular, to a method and device for constructing a materialized view of computing power data of an intelligent computing center. Background Art

[0002] With the development of artificial intelligence technology and computing power technology, the concept of intelligent computing center has emerged. "Intelligent computing center" refers to the use of large-scale heterogeneous computing power resources, including general computing power and intelligent computing power, mainly for artificial intelligence applications (such as artificial intelligence deep learning model development, model training and model reasoning and other scenarios) to provide the required computing power, data and algorithms. Intelligent computing center covers facilities, hardware, software, and can provide full-stack capabilities from bottom-level computing power to top-level application enablement.

[0003] In most data preprocessing processes in intelligent computing centers, distributed file systems are generally used as a centralized data storage to solve various data reading and writing requirements in the data pre-training process. When using distributed file systems to implement data processing in the pre-training process of large language models, it mainly involves data storage, preprocessing, loading and management. Distributed systems can process or store large amounts of data in parallel and are fault-tolerant and highly available.

[0004] However, in the current data preprocessing solutions, whether using distributed file systems or object storage, data processing is limited to traditional storage systems. In the process of data processing, a large amount of data copying and moving work is inevitably required, so that the efficiency of the final data processing will be limited by the metadata and data reading and writing performance of the distributed file system, thereby reducing the efficiency of data processing in the intelligent computing center, limiting the computing power improvement of the intelligent computing center, and resulting in low efficiency in scenarios where efficient data processing is required, such as large language model training and fine-tuning. Summary of the invention

[0005] The embodiments of the present invention provide a method and device for constructing a materialized view of the computing power data of an intelligent computing center, so as to solve the technical problem existing in the related technology that a large amount of data copying and moving work is required in the process of using a traditional distributed file system for data processing, which reduces the efficiency of data processing in the intelligent computing center and limits the improvement of the computing power of the intelligent computing center.

[0006] In order to solve the above-mentioned technical problems, the present invention is achieved as follows:

[0007] In a first aspect, an embodiment of the present invention provides a method for constructing a materialized view of computing power data of an intelligent computing center, the method comprising:

[0008] Step S1: Obtain a data file to be imported into a multi-mode distributed file system;

[0009] Step S2: Acquire feature data of the data file, and save the feature data into a database table of a multi-mode vector database of the multi-mode distributed file system;

[0010] Step S3: obtaining an input target database table identifier, and using the multi-mode distributed file system to obtain the feature data from the target database table, performing preset metadata processing on the feature data, generating metadata, and generating a data materialized view based on the metadata;

[0011] Step S4: Obtain the data directory corresponding to the data materialized view output by the multi-mode distributed file system.

[0012] Optionally, the step S1 includes:

[0013] Step S11: Mount the multi-mode distributed file system;

[0014] Step S12: copy the data file to be imported into the multi-mode distributed file system.

[0015] Optionally, step S2 includes at least one of the following:

[0016] Step S21a: input the data file into a preset Embedding model, output feature data, and save the feature data into a database table of a multi-mode vector database of the multi-mode distributed file system;

[0017] Step S21b: extracting corresponding feature data from the data file, and saving the feature data into a database table of the multi-mode vector database of the multi-mode distributed file system.

[0018] Optionally, step S2 further includes:

[0019] Step S22: Acquire characteristic data of the data file;

[0020] Step S23: Obtain the user's operation instructions, and according to the operation instructions, use the hybrid retrieval capability of the scalar index, vector index and / or full-text index, and / or the multi-way recall capability of the multi-mode vector database of the multi-mode distributed file system to operate on the feature data to obtain processed feature data, and save the processed feature data to the database table of the multi-mode vector database of the multi-mode distributed file system.

[0021] Optionally, the operation includes at least one of the following:

[0022] Filtering the characteristic data based on standard SQL statements;

[0023] Performing format conversion on the feature data based on a standard SQL statement;

[0024] Cleaning the characteristic data based on standard SQL statements;

[0025] The feature data is randomly sorted based on a standard SQL statement.

[0026] Optionally, step S3 includes:

[0027] Step S31: obtaining an input target database table identifier, and determining a target database table from database tables of the multi-mode vector database according to the identifier;

[0028] Step S32: Acquire the feature data from the target database table through an application programming interface.

[0029] Optionally, step S3 further includes:

[0030] Step S33: obtaining the storage path of the input data materialized view;

[0031] Step S34: Store the data directory to the location specified by the storage path.

[0032] Optionally, the method further includes:

[0033] Step S5: Acquire the characteristic data through the data directory;

[0034] Step S6: training a large language model based on the feature data.

[0035] In a second aspect, an embodiment of the present invention provides a device for constructing a materialized view of computing power data of an intelligent computing center, the device comprising:

[0036] An acquisition module, used to acquire data files imported into the multi-mode distributed file system;

[0037] An execution module, configured to obtain feature data of the data file, and save the feature data into a database table of a multi-mode vector database of the multi-mode distributed file system;

[0038] Used to obtain an input target database table identifier, and use the multi-mode distributed file system to obtain the feature data from the target database table, perform preset metadata processing on the feature data, generate metadata, and generate a data materialized view based on the metadata;

[0039] Used to obtain the data directory corresponding to the data materialized view output by the multi-mode distributed file system.

[0040] In a third aspect, an embodiment of the present invention provides an electronic device, comprising: a processor, a memory, and a program stored in the memory and executable on the processor, wherein the program, when executed by the processor, implements the steps of a method for constructing a materialized view of computing power data of an intelligent computing center as described in the first aspect.

[0041] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of a method for constructing a materialized view of computing power data of an intelligent computing center as described in the first aspect are implemented.

[0042] In a fifth aspect, an embodiment of the present invention provides a computer program product, including computer instructions, which, when executed by a processor, implement the steps of a method for constructing a materialized view of computing power data of an intelligent computing center as described in the first aspect.

[0043] In the embodiment of the present invention, the data materialized view directly converts the data generated by the query into a storage format (data directory) without copying or duplicating the data from the original database table. The data materialized view means that the data is pre-calculated and stored, rather than dynamically generated each time it is queried. Therefore, compared with the related art, the efficiency of using a distributed file system for data processing in an intelligent computing center is low, thereby limiting the improvement of the computing power of the intelligent computing center. The use of data materialized views for querying and acquiring data does not require a large amount of data copying and moving work (in the related art, using a distributed file system for data processing requires reading a large amount of data from the distributed file system. These data are usually not stored on a single server, but distributed in multiple physical locations. The distributed file system needs to copy and transfer data directly between different storage locations, which consumes a lot of time and network bandwidth, resulting in a low computing power of the intelligent computing center). It can reduce the time occupied by data processing, save labor costs, and effectively improve the efficiency of data processing, thereby improving the computing power of the intelligent computing center. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. Moreover, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0045] Figure 1 An overall architecture diagram of a method for constructing a materialized view of computing power data of an intelligent computing center provided by an embodiment of the present invention;

[0046] Figure 2A flowchart of a method for constructing a materialized view of computing power data of an intelligent computing center provided by an embodiment of the present invention;

[0047] Figure 3 A flowchart of a method for constructing a materialized view of computing power data of an intelligent computing center provided by an embodiment of the present invention;

[0048] Figure 4 A structural block diagram of a device for constructing a materialized view of computing power data in an intelligent computing center provided by an embodiment of the present invention;

[0049] Figure 5 Schematic diagram of the structure of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0050] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0051] The following first describes the implementation steps, data operations, and applications of a distributed file system in the prior art to clarify the differences between the distributed file system and the multi-mode distributed file system provided by the present invention.

[0052] The following are the implementation steps, data operations, applications and existing problems of distributed systems:

[0053] The steps to achieve this are:

[0054] 1. Data storage: Store raw data (such as text, documents, web pages, etc.) in a distributed file system such as HDFS, GlusterFS or Ceph, and the data is usually stored in the form of chunks to ensure high availability and fault tolerance.

[0055] 2. Data preprocessing: word segmentation and tokenization, that is, breaking down text data into words or subwords and converting them into tokens that the model can understand; data cleaning, that is, removing noise and useless information, such as HTML tags, special characters, etc.; data standardization, that is, unifying the text format, such as lowercase conversion, removing stop words, etc.; data enhancement, that is, expanding data, such as synonym replacement, data augmentation, etc.

[0056] 3. Data loading: Use a distributed computing framework (such as Apache Spark) to process and load data in parallel to ensure efficient data reading and distribution. Batch processing is usually performed when loading data to meet large-scale data processing needs.

[0057] 4. Data management: Manage data metadata and version control to ensure data traceability and consistency. Data sets can be partitioned to improve the efficiency of data processing and access.

[0058] The data operations are:

[0059] 1. Read and write operations: Reading, that is, reading large-scale data in parallel to ensure efficient data flow into the model training process; writing, that is, writing the processed data back to the distributed file system, usually stored in an intermediate format (such as Parquet, Avro) for subsequent processing and analysis.

[0060] 2. Data partitioning and sharding: that is, partitioning and sharding the data to ensure load balancing and efficient processing in a distributed environment.

[0061] 3. Data replication and redundancy: That is, data usually has multiple copies in a distributed file system to ensure high availability and data security.

[0062] Applications of distributed file systems:

[0063] 1. Storing large-scale data: Distributed file systems provide efficient storage solutions that can handle PB (Petabyte)-level data volumes.

[0064] 2. Parallel processing: Integrate with distributed computing frameworks (such as Apache Spark and Hadoop MapReduce) to achieve parallel data processing and computing.

[0065] 3. Fault tolerance and high availability: It can provide redundant data storage and automatic fault recovery to ensure system reliability.

[0066] Main bottlenecks:

[0067] 1. I / O (Input / Output) performance: Read / write speed and bandwidth may become bottlenecks, especially in large-scale parallel read and write operations;

[0068] 2. Data consistency: Maintaining consistency and synchronization between multiple copies of data is a challenge;

[0069] 3. Network bandwidth: The transmission of data between nodes will occupy a lot of network bandwidth, leading to potential network bottlenecks.

[0070] The multi-mode distributed file system provided by the present invention improves the existing distributed file system on the basis of retaining the above advantages and operation procedures of the existing distributed file system. The multi-mode distributed file system refers to a new type of file system that supports unified storage of multi-modal feature data of files and data contained in files on the basis of the basic capabilities of the traditional distributed file system, and provides efficient retrieval and processing capabilities of multi-modal feature data.

[0071] Specifically, a distributed file system based on a distributed database multi-mode vector database as a metadata storage engine, namely a multi-mode distributed file system, can be pre-built. The multi-mode distributed file system can be used as a storage platform for data processing during the pre-training process of a large language model.

[0072] Among them, a multimodal vector database refers to a database that can store and process multiple types of data (such as text, pictures, videos, etc.), provide unified storage and unified processing of multiple modal data (scalars, vectors, key-value pairs, graphs, time series), and provide a unified access interface and query language to achieve complex queries and analysis, and can quickly process large amounts of data.

[0073] Metadata storage engine: Metadata is data about data. In a distributed file system, metadata may include information such as the file's size, location, and creation time. The metadata storage engine is responsible for managing this metadata so that users can quickly find and manage their data.

[0074] A multi-mode distributed file system based on a distributed database combines a multi-mode vector database, a metadata storage engine, and a distributed file system to build a distributed file system. This distributed file system uses a multi-mode vector database as a metadata storage engine and can be called a multi-mode distributed file system. The characteristics of this system are as follows:

[0075] 1. Storage efficiency and performance: The file system itself is distributed and highly scalable. It can store large amounts of data in multiple physical locations. Using a multi-mode vector database as a metadata storage engine can process and access data in the file system faster.

[0076] 2. Simultaneous data processing: When data files are written to the file system, they are not only stored, but users can also analyze and process the data at the same time. The feature data of the file (such as the semantic features of text data) can be directly extracted and saved in the multi-modal vector database, which makes the subsequent query and analysis of these features more efficient.

[0077] In summary, compared with existing distributed file systems, the multi-mode distributed file system has the following differences: after the data file is written into the multi-mode distributed file system, the user can also analyze and process the data at the same time, and save the characteristic data of the data file directly into the metadata storage engine of the multi-mode distributed file system.

[0078] When performing data processing operations during the pre-training and fine-tuning of large language models, users can directly use the multi-mode vector database of the multi-mode distributed file system to filter and shuffle data files. They can also use standardized SQL (Structured Query Language) statements to use the hybrid retrieval and multi-way recall capabilities of scalar indexes, vector indexes, and full-text indexes provided by the multi-mode vector database to quickly process data files. Since the multi-mode vector database itself provides distributed transaction capabilities, users can use distributed transactions and MVCC (Multiversion Concurrency Control) mechanisms to implement flexible and changeable data processing strategies when processing data.

[0079] Among them, scalar index: used to process traditional numerical and categorical data, such as numbers, dates, and enumeration types. Vector index: specially designed for vector data, suitable for storing and retrieving digital representations of unstructured data such as images, text, and voice. Full-text index: used to process large amounts of text data and support complex text queries, including fuzzy matching and keyword searches.

[0080] Multi-Channel Recall: is a technology used to improve data retrieval efficiency and recommendation quality. The main idea of ​​multi-channel recall is to retrieve data simultaneously through multiple different ways or paths, thereby increasing the scope and accuracy of recall.

[0081] After processing the data using standardized SQL statements, you can use the data materialization view capability provided by the multi-mode distributed file system to directly output the reorganized data file collection into a new data set directory. This process does not require data copying and is a built-in function provided by the file system. Compared with the traditional distributed file system, which requires duplication or copying of data to build a new data set, it has a lower time cost and can improve the efficiency of data processing. It is widely used in scenarios that require efficient data processing (such as training of large language models).

[0082] The following is a brief description of the technical terms involved in the present invention.

[0083] The “large language model” mentioned in the present invention refers to a large language model (LLM), which is a language model with a large parameter scale. It is designed to understand and generate human language. It is trained with a large amount of text data and can perform a wide range of tasks including text summarization, translation, sentiment analysis, etc.

[0084] The "computing power" mentioned in the present invention is the ability of computer equipment or computing / data center to process information. It is the ability of computer hardware and software to work together to execute certain computing requirements. It is the computing power to achieve target result output by processing information data. It is a new type of productivity that integrates information computing power, network carrying capacity, and data storage capacity. It mainly provides services to the society through computing power infrastructure.

[0085] The "computing power (CP)" mentioned in the present invention is the ability of a data center server to process data and output results. It is a comprehensive indicator to measure the computing power of a data center, including general computing power, super computing power and intelligent computing power. The commonly used unit of measurement is the number of floating point operations per second (Floating Point Operations Per Second, FLOPS, where 1EFLOPS=10^18FLOPS). The larger the value, the stronger the comprehensive computing power. According to calculations, 1EFLOPS is approximately the computing power output of 5 Tianhe-2A or 500,000 mainstream server CPUs (Central Processing Unit) or 2 million mainstream notebooks. The calculation formula is: CP=CP 通用 +CP 智能 +CP 超级 In the embodiment of the present invention, the computing power adopts the half-precision floating point computing power (FP16) of the graphics card.

[0086] The "carrying capacity" (Network Power, NP) described in the present invention is a manifestation of the data transmission capability of computing facilities, including comprehensive capabilities such as network architecture, network bandwidth, transmission latency, intelligent management and scheduling, etc. Carrying capacity involves network transmission within and between data centers, and is a comprehensive indicator for measuring network transmission scheduling capabilities. In the embodiment of the present invention, the carrying capacity uses the video memory bandwidth. In the embodiment of the present invention, the storage capacity uses the video memory bandwidth.

[0087] The "Storage Power" (SP) described in the present invention refers to the comprehensive capabilities of a data center in terms of data storage capacity, performance, safety and reliability, and green and low-carbon. It is a comprehensive indicator for measuring the data storage capacity of a data center, including external storage devices such as storage arrays and server built-in storage devices. The commonly used unit of measurement for storage capacity is exabyte (EB, 1EB = 2^60bytes), and the commonly used unit of measurement for performance is the number of reads and writes per second per unit capacity (IOPS / TB, Input / Output Operations Per Second / TB). The disaster recovery ratio is an important manifestation of safety and reliability. In the embodiment of the present invention, the storage power adopts the video memory capacity.

[0088] The "computing power infrastructure" described in the present invention is a new type of information infrastructure that integrates information computing power, network carrying capacity, and data storage capacity. It can realize centralized computing, storage, transmission and application of information, and is diverse, ubiquitous, intelligent, agile, safe and reliable.

[0089] The “computing power” mentioned in the present invention includes general computing power CP 通用 、Intelligent computing power CP 智能 And super computing power CP 超级 .

[0090] The "general computing power" mentioned in the present invention refers to the computing power provided by servers based on CPU chips, which is used to support basic general computing such as cloud computing and edge computing.

[0091] The "intelligent computing power" mentioned in the present invention refers to the large-scale deployment of intelligent computing centers based on special chips such as GPU (Graphics Processing Unit), FPGA (Field Programmable Gate Array), ASIC (Application Specific Integrated Circuit) for various innovative artificial intelligence applications, such as natural language processing, machine vision, etc.

[0092] The "super computing power" mentioned in the present invention is mainly the computing power provided by high-performance computing clusters such as supercomputers. It utilizes the centralized computing resources of multiple computer systems working in parallel and uses a dedicated operating system to handle extremely complex or data-intensive problems. It is mainly used for calculations in cutting-edge scientific fields, such as planetary simulation, drug molecule design, and gene analysis.

[0093] The "intelligent computing center" mentioned in the present invention refers to a facility that provides the required computing power, data and algorithms for artificial intelligence applications (such as artificial intelligence deep learning model development, model training and model reasoning) by using large-scale heterogeneous computing resources, including general computing power (CPU) and intelligent computing power (GPU, FPGA, ASIC, etc.). The intelligent computing center covers facilities, hardware, and software, and can provide full-stack capabilities from bottom-level computing power to top-level application enablement.

[0094] The "computing resources" mentioned in the present invention refer to the technologies and facilities with information calculation, transmission, storage and application capabilities required for the development of the digital society, including but not limited to computing resources such as CPUs and GPUs, network resources such as switches and routers, storage resources such as storage arrays and distributed storage, security resources such as firewalls and intrusion detection systems, and supporting and guarantee resources such as wind, fire, water and electricity.

[0095] Please refer to Figure 1 , Figure 1 The overall architecture of a method for constructing a materialized view of computing power data of an intelligent computing center provided by an embodiment of the present invention is shown. Figure 2 A method for constructing a materialized view of computing power data of an intelligent computing center provided by an embodiment of the present invention is shown. Figure 2 The method shown can be applied to Figure 1 As shown in the overall architecture, Figure 2 As shown, the method includes:

[0096] Step S1: Obtain a data file to be imported into a multi-mode distributed file system;

[0097] Step S2: Acquire feature data of the data file, and save the feature data to a database table of a multi-mode vector database of a multi-mode distributed file system;

[0098] Step S3: obtaining an input target database table identifier, and using a multi-mode distributed file system to obtain feature data from the target database table, performing preset metadata processing on the feature data, generating metadata, and generating a data materialized view based on the metadata;

[0099] Step S4: Obtain the data directory corresponding to the data materialized view output by the multi-mode distributed file system.

[0100] In the embodiment of the present invention, the data materialized view directly converts the data generated by the query into a storage format (data directory) without copying or duplicating the data from the original database table. The data materialized view means that the data is pre-calculated and stored, rather than dynamically generated each time it is queried. Therefore, compared with the related art, the efficiency of using a distributed file system for data processing in an intelligent computing center is low, thereby limiting the improvement of the computing power of the intelligent computing center. The use of data materialized views for querying and acquiring data does not require a large amount of data copying and moving work (in the related art, using a distributed file system for data processing requires reading a large amount of data from the distributed file system. These data are usually not stored on a single server, but distributed in multiple physical locations. The distributed file system needs to copy and transfer data directly between different storage locations, which consumes a lot of time and network bandwidth, resulting in a low computing power of the intelligent computing center). It can reduce the time occupied by data processing, save labor costs, and effectively improve the efficiency of data processing work, thereby improving the computing power of the intelligent computing center.

[0101] In a possible implementation, step S1 includes: step S11: mounting the multi-mode distributed file system; step S12: copying the data files to be imported into the multi-mode distributed file system.

[0102] It should be noted that you need to mount the multi-mode distributed file system and import data files first. In specific application scenarios, you can use the Fuse client provided by the multi-mode distributed file system to mount the multi-mode distributed file system on the host where you need to import data. After the mounting is completed, the directory of the multi-mode distributed file system will appear in the directory of the host. On the host where data is being imported, you can use the file copy command of the operating system to transfer the data files to the mount point of the multi-mode distributed file system.

[0103] In a possible implementation, step S2 includes at least one of the following:

[0104] Step S21a: input the data file into a preset Embedding model, output feature data, and save the feature data into a database table of a multi-mode vector database of a multi-mode distributed file system;

[0105] Step S21b: extract corresponding feature data from the data file, and save the feature data to a database table of a multi-mode vector database in a multi-mode distributed file system.

[0106] The Embedding model maps high-dimensional data (such as text) to a low-dimensional space, making the data understandable and processable by computers, making the original data suitable for machine learning and other data analysis tasks. The core of the Embedding model is that it can represent the input data as a point in a continuous numerical space, a process called embedding.

[0107] That is to say, after executing the mounting of the multi-mode distributed file system and the importing operation of the data file, the steps of generating and saving the feature data of the data file can be executed. That is, after the data file is copied, different Embedding models can be used, or the feature data carried by the original source of the data file can be directly read, and the feature data can be directly written into the multi-mode vector database used by the multi-mode distributed file system through standard SQL statements, and stored in the form of a database table.

[0108] In one possible implementation, see Figure 3 , step S2 further includes:

[0109] Step S22: Acquire characteristic data of the data file;

[0110] Step S23: Obtain the user's operation instructions, and according to the operation instructions, use the hybrid retrieval capabilities of the scalar index, vector index and / or full-text index, and / or the multi-way recall capability of the multi-mode vector database of the multi-mode distributed file system to operate on the feature data to obtain processed feature data, and save the processed feature data to the database table of the multi-mode vector database of the multi-mode distributed file system.

[0111] The operation includes at least one of the following: screening feature data based on standard SQL statements; format conversion of feature data based on standard SQL statements; cleaning feature data based on standard SQL statements; and random sorting of feature data based on standard SQL statements.

[0112] In other words, users can use standard SQL statements and take advantage of the hybrid retrieval and multi-way recall capabilities of scalar indexes, vector indexes, and full-text indexes provided by the multi-mode vector database to process, filter, and shuffle related database tables of file feature data in the multi-mode vector database, thereby realizing rapid processing of data files. Specifically, users can directly write SQL statements according to actual needs. After execution, they will obtain a collection list of result data files that need to be used, and save the list in the database table of the multi-mode vector database for subsequent use. For example, a simple SQL statement may be: SELECT*FROM users WHERE age>30. This command means to select (SELECT) all users in the user table (users) who are older than 30 years old.

[0113] In a possible implementation, step S3 includes: step S31: obtaining an input target database table identifier, and determining the target database table from the database table of the multi-mode vector database according to the identifier; step S32: obtaining feature data from the target database table through an application programming interface.

[0114] In a possible implementation, step S3 further includes: step S33: obtaining a storage path of the input data materialized view; step S34: storing the data directory to a location specified by the storage path.

[0115] A data materialized view is a pre-calculated and stored query result set, similar to a physical table, but different from a physical table, it can be updated regularly through the database's internal mechanism. A data materialized view can be viewed as a local copy of remote data, or used to generate a summary table based on the sum of a data table.

[0116] And data materialized views mean that data is pre-calculated and stored, rather than dynamically generated each time a query is made. Therefore, data materialized views can significantly reduce data access and processing time, especially when dealing with complex calculations and large data sets. In the scenario of large language model training, a large amount of training data often needs to be processed and accessed. These data processing steps include but are not limited to text cleaning, word segmentation, tokenization, and feature extraction. If the raw data is processed from scratch every time a large language model is trained, a lot of computing resources and time will be consumed. By adopting data materialized views, the results of frequently performed operations, such as pre-processed data, can be stored. When the data is needed for large language model training, it can be read directly from the data materialized view without having to re-perform complex preprocessing operations. There are several direct benefits to doing this: Reduce latency: Because the data has been pre-processed and stored, the reading speed will be faster; Save resources: Reduce the need for repeated calculations, thereby saving CPU and memory resources; Improve efficiency: Speed ​​up the entire training process, so that large language models can be trained in a shorter time, or more training iterations can be performed in the same time.

[0117] It should be noted that in a multi-mode distributed file system, the API (Application Programming Interface) interface provided by the system can be used to input the data table name of the collection list of previously saved data files and the absolute path (storage path) of the data materialized view that is expected to be generated in the multi-mode distributed file system. The multi-mode distributed file system will generate the metadata required for the data materialized view based on the content of the data table through a series of internal metadata processing, and output it as a data directory in the multi-mode distributed file system.

[0118] In a possible implementation, the method further includes: step S5: acquiring feature data through a data directory; step S6: training a large language model based on the feature data.

[0119] It should be noted that the large language model may be a large language model such as GPT4, Claude-2 or Llama-2.

[0120] When pre-training a large language model, data preparation and processing is a complex and critical process. Data preparation and processing generally includes the following steps and methods:

[0121] 1. Data collection:

[0122] Data sources: Collect large amounts of text data from a variety of sources, which can include the Internet (such as web pages, blogs, forums, etc.), libraries and open access books, research papers and technical documents, social media platforms, and professional domain datasets;

[0123] Data volume: It is necessary to collect as much text data as possible to ensure that the model can capture a wide range of language patterns and knowledge.

[0124] 2. Data cleaning:

[0125] Deduplication: Remove duplicate text data to avoid over-weighting certain information during model training.

[0126] Denoising: Remove useless content such as ads, navigation bars, and footers.

[0127] Filtering: Filter out harmful or inappropriate content to ensure the safety and ethical nature of the model.

[0128] 3. Data preprocessing:

[0129] Word segmentation: Break the text into words or subwords to facilitate model processing. Common methods include using space word segmentation, subword segmentation (such as BPE or WordPiece), etc.

[0130] Tokenization: Convert the segmented text into tokens that the model can understand.

[0131] Standardization: Perform uniform processing on text, such as converting to lowercase, removing special characters, etc.

[0132] Spelling Correction: Correct common spelling errors to improve data quality.

[0133] 4. Data enhancement:

[0134] Data expansion: Expand data volume and increase diversity through synonym replacement, translation, and other methods.

[0135] Data balance: Ensure that different types of data are evenly distributed in the training set to prevent the model from being biased towards certain categories.

[0136] 5. Data segmentation:

[0137] Training set, validation set, test set: Split the dataset into training set, validation set, and test set, usually in a ratio of 80:10:10 (can be set according to actual needs). The training set is used for model training, the validation set is used for hyperparameter tuning, and the test set is used for final evaluation.

[0138] 6. Data formatting:

[0139] Input format: Format the preprocessed text data into the input format required by the model, usually a fixed-length token sequence.

[0140] Feature extraction: In some cases, additional features such as word frequency, syntactic structure, etc. can be extracted to help the model better understand the text.

[0141] 7. Data storage and management storage solutions: Use efficient storage solutions (such as HDF5, TFRecord, etc.) to store processed data for fast reading and training.

[0142] Versioning: Version your datasets to maintain consistency across model improvements and experiments.

[0143] 8. Data review and ethical considerations:

[0144] Data review: The data is rigorously reviewed to ensure that there is no data that violates privacy or other ethical issues.

[0145] Ethical considerations: Ensure the diversity and fairness of the data set to avoid bias in the model in terms of gender, race, etc.

[0146] Through the above steps and methods, a large amount of data can be effectively prepared and processed, providing a solid foundation for the pre-training of large language models. The quality of these steps directly affects the performance and application effect of the model. And in the process of data preprocessing, it is generally necessary to use a distributed storage system to store and read data. Therefore, how to efficiently support various modes and methods of data processing is very important for data processing in the pre-training process of large language models. Based on the multi-mode distributed system provided by the embodiment of the present invention, and the method for constructing a materialized view of computing power data of the intelligent computing center provided by the embodiment of the present invention, the data processing process can be optimized, data processing can be performed efficiently, the training efficiency of large language models can be improved, and the computing power of the intelligent computing center can be improved.

[0147] In a specific application scenario, when pre-training or fine-tuning a large language model, the Fuse client of the multi-mode distributed file system is used to mount the multi-mode distributed file system on the GPU or NPU (Neural-network Processing Unit) machine used for training. After the mounting is completed, the data files in the multi-mode distributed file system can be read on the host directory, and then the previously created data materialized view can be read to load data, and perform training and fine-tuning. By using the data materialized view function provided by the multi-mode distributed file system, the data processing efficiency in the training process of the large language model can be effectively improved, the time occupied by data processing can be reduced, and labor costs can be saved, thereby accelerating the training of the large language model.

[0148] Please refer to Figure 4The embodiment of the present invention further provides a device for constructing a materialized view of computing power data of an intelligent computing center, characterized in that the device 40 includes:

[0149] An acquisition module 41 is used to acquire data files imported into the multi-mode distributed file system;

[0150] An execution module 42 is used to obtain feature data of a data file and save the feature data to a database table of a multi-mode vector database of a multi-mode distributed file system;

[0151] It is used to obtain the input target database table identifier, and use the multi-mode distributed file system to obtain feature data from the target database table, perform preset metadata processing on the feature data, generate metadata, and generate data materialized view based on the metadata;

[0152] Used to obtain the data directory corresponding to the materialized view of data output by the multi-mode distributed file system.

[0153] In a possible implementation, the acquisition module 41 is also used to mount the multi-mode distributed file system; and copy the data files to be imported into the multi-mode distributed file system.

[0154] In one possible implementation, the execution module 42 is also used to input the data file into a preset Embedding model, output feature data, and save the feature data into a database table of a multi-mode vector database of a multi-mode distributed file system; extract corresponding feature data from the data file, and save the feature data into a database table of a multi-mode vector database of a multi-mode distributed file system.

[0155] In a possible implementation, the execution module 42 is further used to obtain characteristic data of the data file;

[0156] Obtain the user's operation instructions, and according to the operation instructions, use the hybrid retrieval capability of the scalar index, vector index and / or full-text index of the multi-mode vector database of the multi-mode distributed file system, and / or the multi-way recall capability to operate on the feature data to obtain processed feature data, and save the processed feature data to the database table of the multi-mode vector database of the multi-mode distributed file system.

[0157] In a possible implementation, the operation includes at least one of the following:

[0158] Filter feature data based on standard SQL statements;

[0159] Convert feature data into a new format based on standard SQL statements;

[0160] Clean the feature data based on standard SQL statements;

[0161] Randomly sort feature data based on standard SQL statements.

[0162] In a possible implementation, the execution module 42 is further used to obtain an input target database table identifier, determine the target database table from the database tables of the multi-mode vector database according to the identifier, and obtain feature data from the target database table through an application programming interface.

[0163] In a possible implementation, the execution module 42 is further configured to obtain a storage path of the input data materialized view; and store the data directory to a location specified by the storage path.

[0164] In a possible implementation, the execution module 42 is further configured to obtain feature data through a data directory and train the large language model based on the feature data.

[0165] In the embodiment of the present invention, the data materialized view directly converts the data generated by the query into a storage format (data directory) without copying or duplicating the data from the original database table. The data materialized view means that the data is pre-calculated and stored, rather than dynamically generated each time it is queried. Therefore, compared with the related art, the efficiency of using a distributed file system for data processing in an intelligent computing center is low, thereby limiting the improvement of the computing power of the intelligent computing center. The use of data materialized views for querying and acquiring data does not require a large amount of data copying and moving work (in the related art, using a distributed file system for data processing requires reading a large amount of data from the distributed file system. These data are usually not stored on a single server, but distributed in multiple physical locations. The distributed file system needs to copy and transfer data directly between different storage locations, which consumes a lot of time and network bandwidth, resulting in a low computing power of the intelligent computing center). It can reduce the time occupied by data processing, save labor costs, and effectively improve the efficiency of data processing work, thereby improving the computing power of the intelligent computing center.

[0166] Please refer to Figure 5 An embodiment of the present invention further provides an electronic device 50, including a processor 51, a memory 52, and a computer program stored in the memory 52 and executable on the processor 51. When the computer program is executed by the processor 51, the various processes based on the above-mentioned embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.

[0167] The embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, each process of the embodiment of the method for constructing a materialized view of computing power data of the intelligent computing center is implemented, and the same technical effect can be achieved. To avoid repetition, it is not repeated here. The computer-readable storage medium is, for example, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0168] An embodiment of the present invention also provides a computer program product, including computer instructions. When the computer instructions are executed by a processor, the various processes of the embodiment of the method for constructing a materialized view of computing power data of the above-mentioned intelligent computing center are implemented, and the same technical effect can be achieved. To avoid repetition, they will not be repeated here.

[0169] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0170] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0171] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation modes, which are merely illustrative rather than restrictive. Under the guidance of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the present invention and the claims, all of which are within the protection of the present invention.

Claims

1. A method for constructing a materialized view of computing power data in an intelligent computing center, characterized in that: include: Step S1: Obtain a data file to be imported into a multi-mode distributed file system; Step S2: Acquire feature data of the data file, and save the feature data into a database table of a multi-mode vector database of the multi-mode distributed file system; Step S3: obtaining an input target database table identifier, and using the multi-mode distributed file system to obtain the feature data from the target database table, performing preset metadata processing on the feature data, generating metadata, and generating a data materialized view based on the metadata; Step S4: Obtain the data directory corresponding to the data materialized view output by the multi-mode distributed file system.

2. The method according to claim 1, characterized in that The step S1 comprises: Step S11: Mount the multi-mode distributed file system; Step S12: copy the data file to be imported into the multi-mode distributed file system.

3. The method according to claim 1, characterized in that The step S2 includes at least one of the following: Step S21a: input the data file into a preset embedding model, output feature data, and save the feature data into a database table of a multi-mode vector database of the multi-mode distributed file system; Step S21b: extracting corresponding feature data from the data file, and saving the feature data into a database table of the multi-mode vector database of the multi-mode distributed file system.

4. The method according to claim 1 or 3, characterized in that: The step S2 further comprises: Step S22: Acquire characteristic data of the data file; Step S23: Obtain the user's operation instructions, and according to the operation instructions, use the hybrid retrieval capability of the scalar index, vector index and / or full-text index, and / or the multi-way recall capability of the multi-mode vector database of the multi-mode distributed file system to operate on the feature data to obtain processed feature data, and save the processed feature data to the database table of the multi-mode vector database of the multi-mode distributed file system.

5. The method according to claim 4, characterized in that The operation includes at least one of the following: Screening the characteristic data based on a standard structured query language SQL statement; Performing format conversion on the feature data based on a standard SQL statement; Cleaning the characteristic data based on standard SQL statements; The feature data is randomly sorted based on a standard SQL statement.

6. The method according to claim 1, characterized in that The step S3 comprises: Step S31: obtaining an input target database table identifier, and determining a target database table from database tables of the multi-mode vector database according to the identifier; Step S32: Acquire the feature data from the target database table through an application programming interface.

7. The method according to claim 1, characterized in that The step S3 further comprises: Step S33: obtaining the storage path of the input data materialized view; Step S34: Store the data directory to the location specified by the storage path.

8. The method according to claim 1, characterized in that: Also includes: Step S5: Acquire the characteristic data through the data directory; Step S6: training a large language model based on the feature data.

9. A device for constructing a materialized view of computing power data in an intelligent computing center, characterized in that: The device comprises: An acquisition module, used to acquire data files imported into the multi-mode distributed file system; An execution module, configured to obtain feature data of the data file, and save the feature data into a database table of a multi-mode vector database of the multi-mode distributed file system; Used to obtain an input target database table identifier, and use the multi-mode distributed file system to obtain the feature data from the target database table, perform preset metadata processing on the feature data, generate metadata, and generate a data materialized view based on the metadata; Used to obtain the data directory corresponding to the data materialized view output by the multi-mode distributed file system.

10. An electronic device, characterized in that: include: A processor, a memory, and a program stored in the memory and executable on the processor, wherein when the program is executed by the processor, the steps of a method for constructing a materialized view of computing power data of an intelligent computing center as described in any one of claims 1 to 8 are implemented.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of a method for constructing a materialized view of computing power data of an intelligent computing center according to any one of claims 1 to 8.

12. A computer program product, characterized in that It includes computer instructions, which, when executed by a processor, implement the steps of a method for constructing a materialized view of computing power data of an intelligent computing center as described in any one of claims 1-8.

Images