Methods, apparatuses, and computing devices for data sharding

Abstract

The application provides a data fragmentation method, which comprises the following steps: obtaining a plurality of stored data; inputting the plurality of data into a sorting model to obtain a score corresponding to each data, wherein the input information of the sorting model comprises the plurality of data, the output information of the sorting model comprises the score corresponding to each data, the sorting model is determined according to the plurality of data and at least one query command, and the query command is used to query target data from the plurality of data; and determining a file block corresponding to each of the plurality of data according to the score corresponding to each data. The method can improve the query speed and efficiency of a database.

Description

Technical Field

[0001] This application relates to the field of databases, and more specifically, to a method, apparatus, and computing device for data sharding. Background Technology

[0002] A database can be understood as a repository for organizing, storing, and managing data according to a data structure. Specifically, a database is a large collection of organized, shareable, and uniformly managed data stored within a computer. In databases, data sharding refers to arranging the data in a specific order on storage (e.g., a disk), that is, the order in which the data in the database is arranged on storage (e.g., a disk).

[0003] In all relevant data sharding schemes, a sorting key needs to be manually specified based on experience, and the storage order of data in the storage module is determined according to the sorting key. In this scheme, the storage order of data in the storage module needs to be determined based on experience, which results in high query costs and low database query speed and efficiency.

[0004] Therefore, improving the query speed and efficiency of databases has become a pressing technical problem that needs to be solved. Summary of the Invention

[0005] This application provides a data sharding method that can improve the query speed and efficiency of a database.

[0006] Firstly, a data sharding method is provided, comprising: acquiring multiple stored data; inputting the multiple data into a sorting model to obtain a score corresponding to each data, wherein the input information of the sorting model includes the multiple data, the output information of the sorting model includes the score corresponding to each data, the sorting model is determined based on the multiple data and at least one query command, the query command being used to query target data from the multiple data; and determining the file block corresponding to each of the multiple data based on the score corresponding to each data.

[0007] In the above technical solution, AI technology is used to adaptively shard the data stored in the memory, thereby guiding the file sharding method corresponding to the stored data. This reduces the query cost and improves the efficiency of data skipping when querying the data in the database again, thus improving the query speed and efficiency of the database.

[0008] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes: obtaining the at least one query command and sample data, the sample data being obtained by sampling the stored plurality of data; and training the ranking model based on the at least one query command and the sample data.

[0009] In the above technical solution, a sorting model is trained based on the query cost of data according to at least one query command. The trained sorting model can be obtained when the query cost is the lowest. The score of each piece of stored data is obtained through the trained sorting model. When data sharding is performed on the multiple pieces of data based on the score, the multiple pieces of data can be organized into file blocks with greater clustering, so that the subsequent query cost is the lowest.

[0010] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes: compressing the multiple data to obtain multiple dimensionality-reduced data; inputting the multiple dimensionality-reduced data into the ranking model to obtain a score corresponding to each data.

[0011] The above technical solution can compress and reduce the dimensionality of the data input to the sorting model, thereby adapting to high-dimensional data.

[0012] In conjunction with the first aspect, in some implementations of the first aspect, the multiple data are input into a compression model to obtain the multiple dimensionality-reduced data, wherein the input information of the compression model includes the multiple data, and the output information of the compression model includes the multiple dimensionality-reduced data.

[0013] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes: training the compression model based on sample data.

[0014] In conjunction with the first aspect, in some implementations of the first aspect, the compression model is a variational autoencoder (VAE).

[0015] In conjunction with the first aspect, in some implementations of the first aspect, the method is applied to a cloud management platform for managing infrastructure that provides cloud services, the infrastructure including at least one cloud data center, each of which is equipped with at least one server for training a ranking model and a compression model.

[0016] Secondly, a data sharding apparatus is provided, comprising: an acquisition module and a processing module, wherein the acquisition module is used to acquire multiple stored data; the processing module is used to input the multiple data into a sorting model to obtain a score corresponding to each data, wherein the input information of the sorting model includes the multiple data, the output information of the sorting model includes the score corresponding to each data, the sorting model is determined based on the multiple data and at least one query command, the query command being used to query target data from the multiple data; the processing module is further used to determine the file blocks corresponding to each of the multiple data based on the score corresponding to each data.

[0017] In conjunction with the second aspect, in some implementations of the second aspect, the apparatus further includes a training module, and the acquisition module is further configured to acquire the at least one query command and sample data, the sample data being obtained by sampling the stored plurality of data; the training module is configured to train the ranking model based on the at least one query command and the sample data.

[0018] In conjunction with the second aspect, in some implementations of the second aspect, the device further includes: a compression module for compressing the multiple data to obtain corresponding multiple dimensionality-reduced data; the processing module is specifically used to input the multiple dimensionality-reduced data into the ranking model to obtain a score corresponding to each data.

[0019] In conjunction with the second aspect, in some implementations of the second aspect, the compression module is specifically used to: input the multiple data into the compression model to obtain the multiple dimensionality-reduced data, wherein the input information of the compression model includes the multiple data, and the output information of the compression model includes the multiple dimensionality-reduced data.

[0020] In conjunction with the second aspect, in some implementations of the second aspect, the training module is also used to train the compressed model based on the sample data.

[0021] In conjunction with the second aspect, in some implementations of the second aspect, the compression model is a variational autoencoder (VAE).

[0022] In conjunction with the second aspect, in some implementations of the second aspect, the device is applied to a cloud management platform for managing infrastructure that provides cloud services, the infrastructure including at least one cloud data center, each of which is equipped with at least one server for training ranking models and compression models.

[0023] It should be understood that the beneficial effects of the second aspect and its various implementation methods correspond to the beneficial effects of the first aspect and its various implementation methods. For details, please refer to the beneficial effects of the first aspect and its various implementation methods, which will not be elaborated here.

[0024] Thirdly, a computing device is provided, including a processor and a memory, and optionally, an input / output interface. The processor controls the input / output interface to send and receive information, the memory stores a computer program, and the processor retrieves and runs the computer program from the memory, causing the program to execute the method of the first aspect or any possible implementation thereof.

[0025] Optionally, the processor can be a general-purpose processor, which can be implemented in hardware or software. When implemented in hardware, the processor can be a logic circuit, integrated circuit, etc.; when implemented in software, the processor can be a general-purpose processor that reads software code stored in memory. This memory can be integrated into the processor or located outside the processor and exist independently.

[0026] Fourthly, a computing device cluster is provided, including at least one computing device, each computing device including a processor and a memory; the processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device, such that the computing device cluster performs the method of the first aspect or any possible implementation thereof.

[0027] Fifthly, a chip is provided that acquires and executes instructions to implement the methods described in the first aspect and any implementation thereof.

[0028] Optionally, as one implementation, the chip includes a processor and a data interface, through which the processor reads instructions stored in the memory and executes the methods in the first aspect and any implementation thereof.

[0029] Optionally, as one implementation, the chip may further include a memory storing instructions, and the processor is used to execute the instructions stored in the memory. When the instructions are executed, the processor is used to perform the method in the first aspect and any implementation thereof.

[0030] In a sixth aspect, a computer program product containing instructions is provided, which, when executed by a computing device, cause the computing device to perform the methods described in the first aspect and any implementation thereof.

[0031] In a seventh aspect, a computer program product containing instructions is provided, which, when run by a cluster of computing devices, causes the cluster of computing devices to perform the methods described in the first aspect and any implementation thereof.

[0032] Eighthly, a computer-readable storage medium is provided, characterized in that it includes computer program instructions, which, when executed by a computing device, perform the method as described in the first aspect and any implementation thereof.

[0033] As examples, these computer-readable storage devices include, but are not limited to, one or more of the following: read-only memory (ROM), programmable ROM (PROM), erasable PROM (EPROM), flash memory, electrically EPROM (EEPROM), and hard drive.

[0034] Alternatively, as one implementation method, the aforementioned storage medium can specifically be a non-volatile storage medium.

[0035] A ninth aspect provides a computer-readable storage medium, characterized in that it includes computer program instructions, which, when executed by a cluster of computing devices, perform the method as described in the first aspect and any implementation thereof.

[0036] As examples, these computer-readable storage devices include, but are not limited to, one or more of the following: read-only memory (ROM), programmable ROM (PROM), erasable PROM (EPROM), flash memory, electrically EPROM (EEPROM), and hard drive.

[0037] Alternatively, as one implementation method, the aforementioned storage medium can specifically be a non-volatile storage medium. Attached Figure Description

[0038] Figure 1 This is a schematic block diagram illustrating the training and inference phases of the AI ​​model applied in embodiments of this application.

[0039] Figure 2 A schematic diagram of an application scenario according to an embodiment of this application is shown.

[0040] Figure 3 This is a schematic flowchart illustrating a data fragmentation method provided in an embodiment of this application.

[0041] Figure 4 This is a schematic block diagram illustrating a method for obtaining data scoring, as provided in an embodiment of this application.

[0042] Figure 5 This is a schematic block diagram illustrating how a compression model and a sorting model are trained based on raw data in a storage module and / or query commands for that raw data, as provided in an embodiment of this application.

[0043] Figure 5 This is a schematic block diagram illustrating a compressed model trained based on raw data, provided in an embodiment of this application.

[0044] Figure 7 This is a schematic block diagram illustrating a method for training a sorting model based on query commands for the original data, as provided in an embodiment of this application.

[0045] Figure 8 This is a schematic diagram illustrating the calculation of query cost for sorted data, provided in an embodiment of this application.

[0046] Figure 9 This is a schematic diagram of a model architecture provided in an embodiment of this application.

[0047] Figure 10 This is a schematic diagram illustrating the optimization of False Postive provided in an embodiment of this application.

[0048] Figure 11 This is a schematic block diagram of a data sharding device 1100 provided in an embodiment of this application.

[0049] Figure 12 This is a schematic diagram of the architecture of a computing device 1500 provided in an embodiment of this application.

[0050] Figure 13 This is a schematic diagram of the architecture of a computing device cluster provided in an embodiment of this application.

[0051] Figure 14 This is a schematic diagram showing the connection between computing devices 1500A and 1500B via a network, as provided in the embodiments of this application. Detailed Implementation

[0052] The technical solutions in this application will now be described with reference to the accompanying drawings.

[0053] This application will present various aspects, embodiments, or features relating to systems comprising multiple devices, components, modules, etc. It should be understood and appreciated that individual systems may include additional devices, components, modules, etc., and / or may not include all devices, components, modules, etc. discussed in conjunction with the accompanying drawings. Furthermore, combinations of these approaches are also possible.

[0054] Furthermore, in the embodiments of this application, the words "exemplary," "for example," etc., are used to indicate that they are examples, illustrations, or descriptions. Any embodiment or design scheme described as "exemplary" in this application should not be construed as being more preferred or advantageous than other embodiments or design schemes. Specifically, the use of the term "exemplary" is intended to present the concept in a concrete manner.

[0055] In the embodiments of this application, "corresponding" and "corresponding" can sometimes be used interchangeably. It should be noted that when the distinction is not emphasized, their intended meanings are consistent.

[0056] The business scenarios described in the embodiments of this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided in the embodiments of this application. As those skilled in the art will know, with the evolution of network architecture and the emergence of new business scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0057] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0058] In this application, "at least one" means one or more, and "more than one" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can mean: A alone, A and B simultaneously, and B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0059] For ease of description, the concepts involved in the embodiments of this application will be explained below.

[0060] 1. Database: A database can be understood as a repository for organizing, storing, and managing data according to a data structure. Specifically, a database is a large collection of organized, shareable, and uniformly managed data stored within a computer. A database can include multiple tablespaces.

[0061] 2. Data sharding: In databases, data sharding refers to arranging data in a specific order on storage (e.g., disk), that is, the order in which the data in the database is arranged on storage (e.g., disk). Common data sharding methods in databases include: A) Random partitioning: Randomly partition the data table into equal-sized partitions.

[0062] B) Range partitioning: Partitioning a data table according to certain rules, such as specifying the range of columns or the time of data insertion.

[0063] C) Segment the data according to the space-filling curve.

[0064] 3. Data skipping Data skipping typically refers to skipping irrelevant data records or data blocks during data reading to reduce the amount of data processed or speed up processing. In databases, this might involve using filters or filtering conditions to select only the necessary data records, or using indexes to skip unnecessary data blocks. In big data processing, data skipping may involve using techniques such as sampling, filtering, or partitioning to reduce the amount of data processed.

[0065] In other words, data skipping is a technique that improves query speed by reducing unnecessary I / O. Data skipping information is automatically collected when data is written. This information (minimum and maximum values) is used during queries to provide faster results. Data skipping is a query optimization with a filter clause that uses statistics to locate the set of data files to query.

[0066] 4. Artificial intelligence (AI) AI (Artificial Intelligence) is the theory, methods, technology, and application systems that utilize digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, artificial intelligence is a branch of computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. Artificial intelligence studies the design principles and implementation methods of various intelligent machines, enabling them to possess perception, reasoning, and decision-making capabilities. Research in the field of artificial intelligence includes robotics, natural language processing, computer vision, decision-making and reasoning, human-computer interaction, recommendation and search, and fundamental AI theories.

[0067] The basic principle of AI is to combine massive amounts of data with powerful computing capabilities and intelligent algorithms to build an AI model that solves specific problems. This AI model can automatically summarize and learn potential patterns or features from the data, thereby achieving a way of thinking that is close to that of humans.

[0068] AI models, also known as AI algorithms (or AI operators), are a collective term for mathematical algorithms built upon the principles of artificial intelligence. They form the foundation for using AI to solve specific problems. Depending on the specific methods and / or technologies used to implement artificial intelligence, AI models can also be called machine learning models, deep learning models, or reinforcement learning models.

[0069] Machine learning is a method for achieving artificial intelligence. Its goal is to design and analyze algorithms (i.e., models) that allow computers to automatically "learn." These designed algorithms are called machine learning models. Machine learning models are algorithms that automatically analyze data to obtain patterns and use these patterns to predict unknown data. Machine learning models are diverse, depending on whether or not the model training relies on the labels corresponding to the training data.

[0070] Deep learning is a new technological field that emerged during machine learning research. Specifically, deep learning is a method in machine learning based on learning deep representations of data. Deep learning interprets data by building neural networks that simulate the human brain's analytical learning process. In the field of AI, deep learning is a learning technique based on deep neural network algorithms. A deep learning model includes an input layer, hidden layers, and an output layer, and it uses multiple nonlinear transformations to process the data. In machine learning methods, almost all features need to be determined by industry experts and then encoded. However, deep learning algorithms attempt to learn features from the data themselves; algorithms designed based on the principles of deep learning are called deep learning models.

[0071] The typical structure of current deep learning models is a deep neural network. A neural network is a mathematical or computational model that mimics the structure and function of biological neural networks (the central nervous system of animals, especially the brain). Neural networks consist of a large number of interconnected neurons performing computations. A neural network can include multiple layers with different functions, each layer containing parameters and computational rules. Depending on the computational formula or function, different layers in a neural network have different names; for example, the layer performing convolution calculations is called a convolutional layer, which is often used for feature extraction from input signals (e.g., images). A neural network can also be composed of multiple sub-neural networks. Different neural network structures can be applied to different scenarios (e.g., classification, recognition) or provide different results when used in the same scenario. The specific differences in neural network structures include one or more of the following: different numbers of network layers, different order of network layers, and different weights, parameters, or computational formulas in each network layer. Various high-accuracy neural networks exist in the industry for applications such as recognition or classification. Some neural networks can be trained on specific datasets and used alone to complete a task or combined with other neural networks (or other functional modules) to complete a task.

[0072] In other words, deep learning models are actually machine learning models with complex neural network structures. Based on whether deep learning models need to rely on the labels of the training data during training, they can also be divided into supervised learning models and unsupervised learning models, which will not be elaborated upon here. Classic deep learning models include convolutional neural networks (CNNs), recurrent neural networks (RNNs), and recursive neural networks (RNNs).

[0073] Before any AI model can be used to solve a specific technical problem, it needs to be trained. AI model training refers to using a specified initial model to compute on training data, and then adjusting the parameters of the initial model based on the computation results, so that the model gradually learns certain patterns and acquires specific functions. Once trained and possessing stable functionality, the AI ​​model can be used for inference. AI model inference is the process of using the trained AI model to compute on input data and obtain predicted inference results.

[0074] like Figure 1As shown, during the training phase, a training set for the deep learning model needs to be constructed based on the objective. The training set includes multiple training data points, each labeled. The label of a training data point represents the correct answer to a specific question, and the label can indicate the objective of training the deep learning model using the training data. For example, to train a deep learning model that can identify different animals, the training set can include images of multiple different animals (i.e., training data). Each image can have a label identifying the type of animal it contains, such as cat or dog. In this example, the type of animal corresponding to each image is the label of that training data.

[0075] When training a deep learning model, training data can be input into the model in batches after parameter initialization. The deep learning model performs calculations (i.e., inference) on the training data to obtain prediction results. The prediction results obtained through inference, along with the corresponding labels of the training data, are used as data for calculating the loss based on the loss function. The loss function is used during the model training phase to calculate the difference (i.e., the loss value) between the model's prediction results on the training data and the labels of that training data. The loss function can be implemented using different mathematical functions; commonly used expressions for loss functions include: mean squared error loss function, logarithmic loss function, least squares method, etc.

[0076] The loss value calculated based on the loss function can be used to update the parameters of a deep learning model. Gradient descent is a common method for updating these parameters. Model training is an iterative process where each iteration infers from different training data and calculates the loss value. The goal of these multiple iterations is to continuously update the parameters of the deep learning model and find the parameter configuration that minimizes or stabilizes the loss value of the loss function.

[0077] During the training phase, to improve training efficiency and post-training model performance, it's necessary to set appropriate hyperparameters. Hyperparameters in deep learning models refer to parameters that cannot be obtained through learning from training data or that cannot be changed by training data; they are a concept relative to the parameters in the model. Hyperparameters of deep learning models are typically set manually based on experience or experiments. These hyperparameters include: learning rate, batch size, and network structure hyperparameters (e.g., number of layers (also called depth), interaction methods between layers, number and size of convolutional kernels, activation functions, etc.). Among these, the learning rate, as a hyperparameter, controls the magnitude of parameter weight updates during training, significantly impacting training speed and accuracy.

[0078] like Figure 1 As shown, a trained deep learning model can be used to infer from input data. In the inference phase, data from real-world application scenarios is typically used as input. The trained deep learning model then infers the results. The inference phase is the practical application of the trained deep learning model, allowing for the rapid use of AI capabilities to solve specific technical problems. Currently, AI has numerous application scenarios, and the inference capabilities of deep learning models can be used in various applications, such as personnel recognition in access control and security systems, video surveillance for violence detection, and express delivery tracking number detection and recognition.

[0079] The above only introduces the training of the most typical deep learning model as an example. The training of other types of models has slight differences, but the principle is similar. Most of them involve inference on the training data and adjusting the parameters in the model based on the inference results, with the goal of obtaining a parameter combination that makes the model performance stable.

[0080] In recent years, the big data industry and cloud computing technology have rapidly developed into a hot topic of interest in academia and industry. Big data has permeated almost all industries, becoming an important factor of production. As the scale of data grows larger and larger, this trend has led to the gradual evolution of traditional database management systems into cloud-native databases and cloud-native data warehouses capable of handling massive amounts of data and responding to queries for terabytes of data in seconds.

[0081] To adapt to larger-scale industry applications, indexing has become a key technology for improving the query capabilities of cloud-native data warehouses. Early cloud-native data warehouses heavily utilized fine-grained indexes (such as B-Trees) to accelerate query processing. However, B-Trees are limited by their large storage size and scalability. In recent years, the trend has shifted towards using scan-oriented data processing strategies, which leverage the high sequential scan bandwidth of modern storage devices. Specifically, cloud-native data warehouses typically distribute data across different cloud storage units using a sharding approach. To facilitate fast retrieval within these cloud storage units, each unit records extreme values ​​for each dimension, enabling rapid filtering and pruning. This method is highly effective for filtering data with a total order. However, multidimensional data lacks a total order; therefore, the currently used random sharding approach often results in data within each storage unit being significantly disparate across multiple dimensions, greatly reducing the effectiveness of filtering using extreme values.

[0082] Existing data sharding methods all require manually specifying the sorting key based on experience. This application provides a data sharding method that uses AI technology to adaptively shard data in a database stored in memory. This reduces query costs and improves data skipping efficiency when querying the database again using a query command, thereby enhancing database query speed and efficiency.

[0083] In one possible implementation, the method provided in this application embodiment can be applied to a cloud scenario, meaning the aforementioned database can be a cloud database. Here, a cloud database refers to a database deployed in a virtual computing environment, offering advantages such as pay-as-you-go pricing, on-demand scaling, high availability, and storage consolidation. The virtual computing environment can refer to a cloud platform. A cloud platform, short for cloud computing platform, provides computing, networking, and storage capabilities based on hardware and software resources. By processing and analyzing massive amounts of data remotely via the network "cloud," it returns the processed data to the user, featuring large-scale, distributed, virtualized, highly available, scalable, on-demand service, and security characteristics. Cloud platforms can achieve rapid deployment and release of configurable computing resources with relatively low management costs or low interaction complexity between users and service providers.

[0084] Figure 2 This illustration shows an application scenario of an embodiment of the present application. The application scenario illustrates a client 210 and a cloud service system, wherein the cloud service system may include one or more server-side components 230. Figure 2 One example is shown, but this application does not limit it. Client 210 is directly or indirectly connected to server 230 via network 220 (wireless or wired network).

[0085] Client 210, deployed on the user side, can act as a cloud database client, providing an interface for using database services. Client 210 can be a virtual machine or a physical machine, such as a desktop computer, laptop, or server. Client 210 can also be software deployed on the user-side device (such as a database client or browser), which is not limited in this application.

[0086] Server 230, acting as the server for the cloud database system, manages the cloud database, such as executing parsing query requests and providing data communication. Server 230 can be a standalone physical server, desktop computer, laptop, or a computing or storage node (also called a data processing device) within a server cluster or distributed system. It provides services to client 210, such as cloud database services. Specifically, server 230 can run databases such as PostgreSQL, MySQL, Oracle Database, and GaussDB (for Influx) for data storage and management, enabling convenient operations such as adding, deleting, viewing, and modifying data.

[0087] Client 210 can communicate with server 230 via network 220, where network 220 uses standard communication technologies and / or protocols. Network 220 includes, but is not limited to, any combination of data center networks, storage area networks (SANs), local area networks (LANs), metropolitan area networks (MANs), wide area networks (WANs), mobile, wired or wireless networks, private networks, or virtual private networks. In some implementations, technologies and / or formats including Hypertext Markup Language (HTML), Extensible Markup Language (XML), etc., are used to represent data exchanged over the network. Furthermore, conventional encryption technologies such as Secure Sockets Layer (SSL), Transport Layer Security (TLS), Virtual Private Network (VPN), and Internet Protocol Security (IPsec) can be used to encrypt all or part of the link. In other embodiments, custom and / or dedicated data communication technologies can be used to replace or supplement the aforementioned data communication technologies.

[0088] Users can send data access requests to server 230 through client 210 to access data in server 230. Processes in server 230 can respond to received data access requests by performing a series of read and / or write operations to achieve data storage and management.

[0089] Let's combine the following... Figure 3 This application provides a detailed description of a data fragmentation method according to embodiments. It should be understood that... Figure 3 The examples are merely to help those skilled in the art understand the embodiments of this application, and are not intended to limit the embodiments of the application to... Figure 3 The specific numerical values ​​or specific scenarios illustrated. Those skilled in the art will understand based on... Figure 3 The examples given below can obviously be modified or varied in various ways, and such modifications and variations also fall within the scope of the embodiments of this application.

[0090] Figure 3 This is a schematic flowchart illustrating a data fragmentation method provided in an embodiment of this application. Figure 3 As shown, the method may include steps 310-330, which will be described in detail below.

[0091] Step 310: Retrieve the original data stored in the database from the storage module.

[0092] As an example, in this embodiment of the application, raw data from a database stored on a storage module (e.g., a disk) can be obtained. Figure 4 As shown, the raw data is data that the user has already stored in the storage module.

[0093] Specifically, the storage module (e.g., a disk) includes multiple data blocks, each containing multiple raw data stored by the user in the storage module.

[0094] Optionally, in some embodiments, the storage module described above may be an object storage.

[0095] It should be understood that data partitioning involves dividing data into multiple file blocks according to certain rules, each of which can be processed or manipulated independently. Data partitioning is commonly used to optimize algorithms, improve processing efficiency, and reduce unnecessary computation and data transfer. In databases, data partitioning is also known as data partitioning, which involves distributing data across different physical locations or logical partitions according to certain rules to improve data access performance and management capabilities. In distributed systems, data partitioning can also be used to balance load and improve system scalability. In databases, data partitioning typically needs to consider factors such as data access frequency, data size, and system load. In distributed systems, data partitioning also needs to consider issues such as load balancing, data replication, and fault recovery.

[0096] The aforementioned file block can also be called a data block.

[0097] It should also be understood that in a database, data is primarily organized in tabular form. Therefore, data in a database can also be called tabular data, and each piece of data can be a row in a table.

[0098] Step 320: Input the acquired raw data into the sorting model to obtain a score for each raw data point.

[0099] In this embodiment of the application, after obtaining the raw data of the database stored on the storage module (e.g., disk), such as Figure 4 As shown, the raw data can be input into a pre-trained ranking model, which can output a score for each piece of raw data. Specifically, the acquired raw data can be used as input information for the ranking model, and the score for each piece of raw data can be used as output information for the ranking model.

[0100] Step 330: Determine the file block corresponding to each piece of raw data based on the score of each piece of raw data.

[0101] In this embodiment of the application, after obtaining the score of each original data stored in the storage module, the multiple original data can be divided into data blocks in descending order of their scores, that is, the file block corresponding to each data in the original data can be determined.

[0102] Specifically, since the size of each file block is fixed, once the scores of all the raw data are obtained, the method for allocating file blocks for the entire raw data is determined. Therefore, as... Figure 4 As shown, based on the size of the data that can be included in each file block, and according to the score of each original data, the multiple original data can be re-divided into their respective file blocks.

[0103] In this embodiment of the application, after dividing the original data stored in the storage module into their respective file blocks, it is also necessary to update the information of each file block, such as the maximum value (max) and minimum value (min) of the data maintained by each file block.

[0104] In the above technical solution, since the sorting model is a pre-trained sorting model that minimizes the query cost, the scoring of the original data obtained through this sorting model can obtain the optimal data sharding method, which can minimize the query cost when subsequent query commands query the reordered data, thereby improving the efficiency of data skipping during data query and improving the query speed and efficiency of the database.

[0105] Optionally, in some embodiments, to improve the inference efficiency of the ranking model, the obtained original data can be dimensionality-reduced, compressing the high-dimensional original data into low-dimensional data. This low-dimensional data is used as input to the ranking model, and the output of the ranking model is a score for this low-dimensional data. Based on this score, the optimal data sharding method for the original data can be obtained. In this way, the dimensionality of the input information to the ranking model is reduced, thereby improving the efficiency of the ranking model in obtaining scores. Furthermore, the compression model improves the model's ability to adapt to high-dimensional data; it can naturally filter high-dimensional data, alleviating the problem of the curse of dimensionality. It should be understood that the curse of dimensionality refers to the weakening or even disappearance of spatial proximity when the dimensionality is too high, making it impossible to achieve the desired data sharding effect using traditional spatial indexing structures.

[0106] One possible implementation is, such as Figure 4 As shown, the original data can be reduced in dimensionality using a compression model to obtain low-dimensional data, which is then output to a pre-trained ranking model. This ranking model can output a score for each piece of original data.

[0107] This application does not impose specific limitations on the compression model, as long as it can compress high-dimensional data into low-dimensional data. For example, a variational autoencoder (VAE) can be used as the compression model.

[0108] It should be understood that a Generative Evolution (VAE) is a type of model that combines the advantages of deep learning and probabilistic models. It learns the data distribution and generates new data samples through an encoder (also called a latent variable model) and a decoder (also called a generator). The core idea of ​​a VAE is to use a neural network to learn the probability distribution of latent variables, which are an intermediate step in data generation. The neural network is trained to maximize the likelihood probability of the data while keeping the distribution of the latent variables close to a specified simple distribution (such as the standard normal distribution). During training, the VAE uses variational inference and reparameterization techniques to optimize the objective function. It first uses an encoder (also called a latent variable model) to encode the input data into latent variables in the latent space, and then uses a decoder (also called a generator) to decode the latent variables back into the original data. After training, the VAE can generate new data samples by sampling new latent variables from the latent space and decoding them. These new data samples are similar in data distribution to the training data, but they are entirely new and unseen data.

[0109] The following is combined with Figures 5 to 7 ,right Figure 4 The training process of the compression model and the ranking model in this paper is described in detail. It should be understood that... Figures 5 to 7The examples are merely to help those skilled in the art understand the embodiments of this application, and are not intended to limit the embodiments of the application to... Figures 5 to 7 The specific numerical values ​​or specific scenarios illustrated. Those skilled in the art will understand based on... Figures 5 to 7 The examples given below can obviously be modified or varied in various ways, and such modifications and variations also fall within the scope of the embodiments of this application.

[0110] As an example, such as Figure 5 As shown, in this embodiment of the application, the compression model and the sorting model can be trained based on the original data in the storage module and / or the query command on the original data, so that the compression model can achieve accurate data compression of the input high-dimensional data, and the sorting model can score the input data and re-sort the input data according to the score, so that the query cost or scanning cost is minimized.

[0111] For example, Figure 6 This is a schematic block diagram illustrating a compressed model trained based on raw data, provided in an embodiment of this application. For example... Figure 6 As shown, taking VAE as an example, VAE includes an encoder and a decoder. The dataset (sampled data obtained by sampling the original data stored in the storage module) is used as input to the encoder. The encoder compresses the data in the dataset into a low-dimensional Gaussian distribution X through the encoder network. Specifically, assuming the data in the dataset is a high-dimensional vector, denoted by T (T=[d1,…,dn]), this high-dimensional vector T is processed by the encoder neural network, X = F_encode(T), to obtain the low-dimensional Gaussian distribution X. The mean and variance of each dimension of the low-dimensional Gaussian distribution are... Next, latent variables Z are sampled from the multidimensional Gaussian distribution X. It should be understood that Z is the low-dimensional data obtained by compressing the high-dimensional data in the dataset.

[0112] In this embodiment, to achieve accurate compression of high-dimensional data by the encoder, the Z obtained by the encoder compression needs to be input to the decoder to obtain the decoded input T'. The VAE attempts to fit T', i.e., reconstruct T. The loss function of the VAE consists of two parts: reconstruction loss and KL divergence loss. The encoder and decoder parameters included in the VAE are continuously adjusted according to this loss function until the loss function of the VAE meets the requirements, at which point the VAE can be considered to have completed the training process.

[0113] It should be understood that reconstruction loss measures the difference between the samples generated by the decoder and the original samples, while KL divergence loss measures the difference between the distribution in the latent space and the standard normal distribution.

[0114] For example, Figure 7This is a schematic block diagram illustrating a method for training a ranking model based on query commands for the original data, as provided in an embodiment of this application. Figure 7 As shown, the input data is fed into the ranking model, which outputs a score for each input data point, for example, S1-Sn. Since the scores for the input data are not differentiable when sorting, this prevents subsequent loss calculation. Therefore, this embodiment uses a softrank operator to convert the scores into ranking indices, i.e., converting each score into a corresponding ranking value, for example, converting S1-Sn into R1-Rn. Based on the size of R1-Rn and the file block configuration size, the file blocks corresponding to the input data in R1-Rn are determined, thus achieving one partitioning of the file blocks to which the input data belongs. Then, the statistical information of each file block (i.e., Min and Max values) is recorded. Based on the query conditions, it is determined whether the file block needs to be scanned, obtaining the query cost. The parameters of the ranking model are continuously adjusted based on the query cost, thereby continuously adjusting the current input data ranking until the calculated query cost is minimized. At this point, the ranking model can be considered to have completed the training process, and the ranking model can be used for inference (for...). Figure 3 (Sort the original data in the data).

[0115] It should be understood that the Soft rank operator uses ranking operations to transform the problem into a linear programming problem on a permutation surface, and uses regularization terms to enable the Soft rank operator to compute the sort index and make it differentiable.

[0116] It should be noted that the embodiments in this application do not specifically limit the input data of the ranking model. For example, the input data of the ranking model can directly be... Figure 6 The dataset in the image (sampled data obtained by sampling the original data stored in the storage module), that is, the input data of the ranking model is uncompressed high-dimensional data. Another example is that the input data of the ranking model can also be... Figure 6 The data in the dataset is compressed to reduce its dimensionality (e.g., the latent variable Z). In other words, the input data of the ranking model is the low-dimensional data after compressing the high-dimensional data.

[0117] For example, Figure 8 This is a schematic diagram illustrating the calculation of query cost for sorted data, provided in an embodiment of this application. For example... Figure 8As shown, the boxes represent the range to be scanned and queried based on the query command. Here, three blocks are covered by the query command: B2, B3, and B4. Since the data blocks have been determined, this embodiment can retrieve all the data that needs to be scanned and obtain its corresponding rank. Figure 8 In this example, assuming a block has a fixed size of 5 (i.e., a block contains 5 data items), then Rank 5 (R5) to Rank 20 (R20) all need to be scanned. The sorting difference value of each block is calculated using the following formula, thus obtaining the final query cost (ScanCost). Finally, in this embodiment, a gradient descent algorithm can be used to minimize the calculated ScanCost, thereby continuously adjusting the current data partitioning method until the ScanCost is minimized.

[0118]

[0119] This application provides a function to automatically adjust the data sharding method on disk. Based on the query commands and data provided by the user, advanced artificial intelligence methods (e.g., a trained sorting model) are used to automatically select the optimal data sharding method on disk without administrator intervention. By using automation to adjust table sharding, the fastest query performance can be achieved without investing time in manually adjusting and implementing table optimization.

[0120] In some embodiments, if the new data sharding method determined in this application can improve cluster performance, the table will be automatically changed within hours after the cluster is created, with minimal impact on queries.

[0121] For example, Figure 9 This is a specific model architecture provided in an embodiment of this application, which can correspond to the compression model or the sorting model described above. For example... Figure 9 As shown, this model architecture is the Attention Calculation model architecture, which is a model structure used for calculating the Attention mechanism.

[0122] It should be understood that the Attention mechanism is a widely used technique in deep learning tasks, especially in fields such as natural language processing, image recognition, and speech recognition. It is used to combine input data with the target task to better capture important features and patterns in the data. The Attention mechanism refers to improving the response to important features by learning a set of weights to weight features at different scales and semantics. Intuitively, the Z-order value of high-dimensional indexing and the attention network in deep learning seem like two completely different concepts. Z-order values ​​are used for indexing and fast retrieval of spatial data, while attention networks are used to capture dependencies in sequential data. However, if we consider their roles and goals abstractly, we can find some conceptual similarities: dimensionality compression versus information focusing: Z-order values ​​compress high-dimensional spatial information into one dimension to enable efficient retrieval and computation in low-dimensional space. The attention mechanism, on the other hand, selectively focuses on the most critical information (which we learn from both the load and the data) in a high-dimensional information flow, compressing the information into a more useful form.

[0123] Attention Calculation model architecture typically includes the following components: 1. Input layer: Responsible for receiving raw input data, such as text, images, or audio.

[0124] 2. Encoder: Converts the input data into a vector representation for subsequent processing.

[0125] 3. Decoder: Decodes the vector representation output by the encoder into meaningful output data.

[0126] 4. Attention Mechanism: Introducing an attention mechanism between the encoder and decoder helps the model better capture important features and patterns in the input data.

[0127] 5. Output layer: Converts the data output by the decoder into the final output result.

[0128] In the Attention Calculation model architecture, the computation methods for the Attention mechanism typically include dot product, additive, and multilayer perceptron. Dot product Attention is further divided into Scaled Dot-Product Attention and Multi-Head Attention. Additive Attention includes Additive Attention and Location-Sensitive Attention. Multilayer perceptron Attention includes Multi-Head Attention in the Transformer model.

[0129] In summary, the core idea of ​​the Attention Calculation model architecture is to combine input data with the target task and introduce an attention mechanism to better capture important features and patterns in the data, thereby improving the model's performance and accuracy.

[0130] For example, Figure 10 This is a diagram illustrating the optimization of False Positive. For example... Figure 10 The image illustrates a visual description of a database query result, specifically an optimization of data block scanning based on min-max values. Each block consists of two rows of data, and the minimum and maximum values ​​for each block are given. These min-max values ​​allow the query engine to decide whether to skip a block, thus improving query efficiency. The query condition is to select records where the age is between 55 and 65 and the savings are greater than 50. Theoretically, using the block's min-max value, the query engine can skip some blocks that do not meet the condition. However, in this example, because the min-max range of each block overlaps with the query condition, the query engine cannot determine which block to skip, thus requiring a scan of each block. More specifically, let's consider the second block (IDs 3 and 4). Although the data within this block does not meet the query condition (e.g., ID 4's age is 80, exceeding the 55-65 range), because the age range of this block (40-80) overlaps with the query condition, the query engine cannot determine that the data in the entire block does not meet the condition, and therefore cannot skip this block. This is a "false positive" situation. Here, "false positive" means that we mistakenly believe a block might contain data that satisfies the query criteria, even when it may not. This causes the query engine to scan certain blocks unnecessarily, reducing query efficiency.

[0131] We propose a high-dimensional index based on a learnable method, where a machine learning model is used to predict the ranking value of each row in the database. After determining which data blocks need to be queried, we extract the row IDs corresponding to the min-max values ​​of each column from each scanned block. Using the idea of ​​contrastive learning, we train the model to make its predicted IDs as far away as possible from the IDs of these scanned blocks, thereby learning how to skip these blocks in future queries to improve query efficiency.

[0132]

[0133] The numerator is the distance between line r1 and the point ri that needs to be moved away (the interval between the two points coincides with the query), and the denominator is the distance between point r1 and the point rk that is close to it (the interval between the two points does not coincide with the query).

[0134] The above text combined Figures 1 to 10 The present application describes in detail the data fragmentation method provided in the embodiments. The following will combine... Figures 11-14 The embodiments of the apparatus of this application are described in detail below. It should be understood that the descriptions of the method embodiments correspond to the descriptions of the apparatus embodiments; therefore, any parts not described in detail can be referred to the foregoing method embodiments.

[0135] Figure 11 This is a schematic block diagram of a data fragmentation device 1100 provided in an embodiment of this application. The device 1100 can be implemented by software, hardware, or a combination of both. The device 1100 provided in this embodiment can implement the embodiments of this application. Figure 3 The method flow shown includes an apparatus 1100 comprising an acquisition module 1110 and a processing module 1120. The acquisition module 1110 is used to acquire multiple stored data. The processing module 1120 is used to input the multiple data into a sorting model to obtain a score corresponding to each data. The input information of the sorting model includes the multiple data, and the output information of the sorting model includes the score corresponding to each data. The sorting model is determined based on the multiple data and at least one query command, which is used to query target data from the multiple data. The processing module 1120 is also used to determine the file block corresponding to each of the multiple data based on the score corresponding to each data.

[0136] Optionally, the device 1100 further includes a training module, and the acquisition module 1110 is further configured to acquire the at least one query command and sample data, wherein the sample data is obtained by sampling the stored plurality of data; the training module is configured to train the ranking model based on the at least one query command and the sample data.

[0137] Optionally, the device 1100 further includes a compression module for compressing the multiple data to obtain multiple dimensionality-reduced data; the processing module 1120 is specifically used to input the multiple dimensionality-reduced data into the ranking model to obtain a score corresponding to each data.

[0138] Optionally, the compression module is specifically used to: input the multiple data into the compression model to obtain the multiple dimensionality-reduced data, wherein the input information of the compression model includes the multiple data, and the output information of the compression model includes the multiple dimensionality-reduced data.

[0139] Optionally, the training module is also used to train the compressed model based on the sample data.

[0140] Optionally, the compression model is a variational autoencoder (VAE).

[0141] Optionally, the device 1100 is applied to a cloud management platform for managing infrastructure that provides cloud services, the infrastructure including at least one cloud data center, each cloud data center having at least one server for training ranking and compression models.

[0142] The device 1100 here can be embodied in the form of a functional module. The term "module" here can be implemented in software and / or hardware, without specific limitation.

[0143] For example, a "module" can be a software program, a hardware circuit, or a combination of both that implements the above functions. For instance, the implementation of module 1110 will be described below using module 1110 as an example. Similarly, the implementation of other modules, such as processing module 1120, compression module, and training module, can refer to the implementation of module 1110.

[0144] As an example of a software functional unit, the acquisition module 1110 may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, or a container. Further, the aforementioned computing instance may be one or more. For example, the acquisition module 1110 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region or in different regions. Further, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including one or more geographically proximate data centers. Typically, a region may include multiple AZs.

[0145] Similarly, multiple hosts / virtual machines / containers used to run this code can be distributed within the same Virtual Private Cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Communication between two VPCs within the same region, as well as between VPCs in different regions, requires a communication gateway to be set up within each VPC to enable interconnection between VPCs.

[0146] As an example of a hardware functional unit, the acquisition module 1110 may include at least one computing device, such as a server. Alternatively, the acquisition module 1110 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be implemented using a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), generic array logic (GAL), or any combination thereof.

[0147] The multiple computing devices included in the acquisition module 1110 can be distributed in the same region or in different regions. Similarly, the multiple computing devices included in the acquisition module 1110 can be distributed in the same Availability Zone (AZ) or in different AZs. Likewise, the multiple computing devices included in the acquisition module 1110 can be distributed in the same Virtual Private Cloud (VPC) or in multiple VPCs. These multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0148] Therefore, the modules of the various examples described in the embodiments of this application can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0149] It should be noted that the device provided in the above embodiments is only illustrated by the division of the functional modules described above when executing the above methods. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. For example, the acquisition module 1110 can be used to execute any step in the above methods, the processing module 1120 can be used to execute any step in the above methods, the compression module can be used to execute any step in the above methods, and the training module can be used to execute any step in the above methods. The steps implemented by the acquisition module 1110, the processing module 1120, the compression module, and the training module can be specified as needed. By implementing different steps in the above methods through the acquisition module 1110, the processing module 1120, the compression module, and the training module, all the functions of the above device can be realized.

[0150] Furthermore, the apparatus and method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process can be found in the method embodiments above, which will not be repeated here.

[0151] The method provided in this application can be executed by a computing device, which can also be referred to as a computer system. It includes a hardware layer, an operating system layer running on top of the hardware layer, and an application layer running on the operating system layer. The hardware layer includes hardware such as processing units, memory, and memory control units; the functions and structure of this hardware will be described in detail later. The operating system is any one or more computer operating systems that implement business processing through processes, such as Linux, Unix, Android, iOS, or Windows. The application layer includes applications such as browsers, address books, word processing software, and instant messaging software. Optionally, the computer system can be a handheld device such as a smartphone, or a terminal device such as a personal computer; this application does not particularly limit this, as long as the method provided in this application can be used. The executing entity of the method provided in this application can be a computing device, or a functional module within the computing device capable of calling and executing programs.

[0152] The following is combined with Figure 12 This application provides a detailed description of a computing device provided in an embodiment.

[0153] Figure 12 This is a schematic diagram of the architecture of a computing device 1500 provided in an embodiment of this application. The computing device 1500 can be a server, a computer, or other device with computing capabilities. Figure 12 The computing device 1500 shown includes at least one processor 1510 and a memory 1520.

[0154] It should be understood that this application does not limit the number of processors and memories in the computing device 1500.

[0155] The processor 1510 executes instructions in the memory 1520, causing the computing device 1500 to implement the method provided in this application. Alternatively, the processor 1510 executes instructions in the memory 1520, causing the computing device 1500 to implement the various functional modules provided in this application, thereby implementing the method provided in this application.

[0156] Optionally, the computing device 1500 also includes a communication interface 1530. The communication interface 1530 uses a transceiver module, such as, but not limited to, a network interface card or a transceiver, to enable communication between the computing device 1500 and other devices or communication networks.

[0157] Optionally, the computing device 1500 also includes a system bus 1540, wherein the processor 1510, memory 1520, and communication interface 1530 are respectively connected to the system bus 1540. The processor 1510 can access the memory 1520 through the system bus 1540; for example, the processor 1510 can perform data read / write or code execution in the memory 1520 through the system bus 1540. The system bus 1540 is a peripheral component interconnect express (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The system bus 1540 is divided into an address bus, a data bus, a control bus, etc. For ease of illustration, Figure 12 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0158] In one possible implementation, the processor 1510 primarily functions to interpret the instructions (or code) of a computer program and process data within the computer software. The instructions of the computer program and the data within the computer software can be stored in memory 1520 or cache 1516.

[0159] Optionally, processor 1510 may be an integrated circuit chip with signal processing capabilities. By way of example and not limitation, processor 1510 can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor is a microprocessor, etc. For example, processor 1510 is a central processing unit (CPU).

[0160] Optionally, each processor 1510 includes at least one processing unit 1512 and a memory control unit 1514.

[0161] Optionally, the processing unit 1512, also known as the core, is the most important component of the processor. The processing unit 1512 is manufactured from single-crystal silicon using a specific production process. All calculations, command reception, command storage, and data processing are performed by the core. Each processing unit independently executes program instructions, utilizing parallel computing capabilities to accelerate program execution. Each processing unit has a fixed logical structure; for example, a processing unit includes logical units such as a Level 1 cache, a Level 2 cache, an execution unit, an instruction-level unit, and a bus interface.

[0162] In one implementation example, the memory control unit 1514 controls the data interaction between the memory 1520 and the processing unit 1512. Specifically, the memory control unit 1514 receives memory access requests from the processing unit 1512 and controls access to memory based on the memory access requests. By way of example and not limitation, the memory control unit is a device such as a memory management unit (MMU).

[0163] In one implementation example, each memory control unit 1514 addresses the memory 1520 via the system bus. An arbitrator is configured in the system bus. Figure 12 (Not shown in the image), this arbitrator is responsible for handling and coordinating competing accesses by multiple processing units 1512.

[0164] In one implementation example, the processing unit 1512 and the memory control unit 1514 are connected via internal chip connection lines, such as address lines, thereby enabling communication between the processing unit 1512 and the memory control unit 1514.

[0165] Optionally, each processor 1510 also includes a cache 1516, which is a buffer for data exchange (called a cache). When the processing unit 1512 needs to read data, it first looks for the required data in the cache. If the data is found, it is executed directly; otherwise, it looks for the data in memory. Since the cache operates much faster than memory, its role is to help the processing unit 1512 run faster.

[0166] The memory 1520 provides runtime space for processes in the computing device 1500. For example, the memory 1520 stores the computer program (specifically, the program code) used to generate the process. After the computer program is executed by the processor to generate a process, the processor allocates corresponding storage space for the process in the memory 1520. Furthermore, the aforementioned storage space further includes text segments, initialized data segments, bit initialized data segments, stack segments, heap segments, etc. The memory 1520 stores data generated during the process's execution, such as intermediate data or process data, in the aforementioned process-specific storage space.

[0167] Optionally, the memory, also known as RAM, is used to temporarily store the data processed by the processor 1510, as well as data exchanged with external storage devices such as hard disks. As long as the computer is running, the processor 1510 will load the data that needs to be processed into RAM for processing, and after the processing is completed, the processing unit 1512 will send the result out.

[0168] By way of example and not limitation, memory 1520 is volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. Non-volatile memory is read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory is random access memory (RAM) used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct rambus RAM (DRRAM). It should be noted that the memory 1520 of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0169] The above-described structure of the computing device 1500 is merely illustrative and is not intended to limit the application. The computing device 1500 in this application includes various hardware components found in existing computer systems. For example, the computing device 1500 may also include other memories besides the memory 1520, such as disk storage. Those skilled in the art should understand that the computing device 1500 may also include other devices necessary for normal operation. Furthermore, depending on specific needs, those skilled in the art should understand that the computing device 1500 may also include hardware devices for implementing other additional functions. Moreover, those skilled in the art should understand that the computing device 1500 may only include the devices necessary for implementing the embodiments of this application, and may not necessarily include... Figure 12 All the devices shown.

[0170] This application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device may be a server. In some embodiments, the computing device may also be a desktop computer, a laptop computer, or a smartphone, or other terminal device.

[0171] like Figure 13 As shown, the computing device cluster includes at least one computing device 1500. The memory 1520 of one or more computing devices 1500 in the computing device cluster may store the same instructions for performing the methods described above.

[0172] In some possible implementations, the memory 1520 of one or more computing devices 1500 in the computing device cluster may also each store a portion of the instructions for executing the above-described methods. In other words, a combination of one or more computing devices 1500 can jointly execute the instructions of the above-described methods.

[0173] It should be noted that the memory 1520 in different computing devices 1500 within the computing device cluster can store different instructions, each used to execute a portion of the functions of the aforementioned device. That is, the instructions stored in the memory 1520 of different computing devices 1500 can implement the functions of one or more modules within the aforementioned device.

[0174] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. This network can be a wide area network (WAN) or a local area network (LAN), etc. Figure 14 One possible implementation is shown. For example... Figure 14 As shown, the two computing devices 1500A and 1500B are connected via a network. Specifically, they are connected to the network through the communication interfaces in each computing device.

[0175] It should be understood that Figure 14 The functions of computing device 1500A shown can also be performed by multiple computing devices 1500. Similarly, the functions of computing device 1500B can also be performed by multiple computing devices 1500.

[0176] In this embodiment, a computer program product containing instructions is also provided. The computer program product may be a software or program product containing instructions capable of running on a computing device or stored on any usable medium. When run on a computing device, it causes the computing device to perform the methods provided above, or causes the computing device to perform the functions of the apparatus provided above.

[0177] In this embodiment, a computer-readable storage medium is also provided. This computer-readable storage medium can be any available medium that a computing device can store, or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that, when executed on a computing device, cause the computing device to perform the method described above.

[0178] It should be understood that in the various embodiments of this application, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0179] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0180] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0181] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0182] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0183] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0184] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0185] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for data fragmentation, characterized in that, The method includes: Retrieve multiple data entries from storage; The multiple data points are input into a ranking model to obtain a score corresponding to each data point. The input information of the ranking model includes the multiple data points, and the output information of the ranking model includes the score corresponding to each data point. The ranking model is determined based on the multiple data points and at least one query command, which is used to query target data from the multiple data points. Based on the score corresponding to each of the data, the file blocks corresponding to each of the multiple data are determined.

2. The method according to claim 1, characterized in that, The method further includes: Obtain the at least one query command and sample data, wherein the sample data is obtained by sampling the stored plurality of data; The ranking model is trained based on the at least one query command and the sample data.

3. The method according to claim 1 or 2, characterized in that, The method further includes: The multiple data sets are compressed to obtain corresponding multiple dimensionality-reduced data sets; The step of inputting the multiple data points into a sorting model to obtain a score corresponding to each data point includes: The multiple dimensionality-reduced data are input into the ranking model to obtain a score corresponding to each data point.

4. The method according to claim 3, characterized in that, The compression process of the multiple data sets to obtain corresponding multiple dimensionality-reduced data sets includes: The multiple data are input into a compression model to obtain the multiple dimensionality-reduced data, wherein the input information of the compression model includes the multiple data, and the output information of the compression model includes the multiple dimensionality-reduced data.

5. The method according to claim 4, characterized in that, The method further includes: The compression model is obtained by training based on the sample data.

6. The method according to claim 4 or 5, characterized in that, The compression model is a variational autoencoder (VAE).

7. The method according to any one of claims 1 to 6, characterized in that, The method is applied to a cloud management platform for managing infrastructure that provides cloud services. The infrastructure includes at least one cloud data center, and each cloud data center is equipped with at least one server for training a ranking model and a compression model.

8. A data sharding apparatus, characterized in that, The device includes: The acquisition module is used to retrieve multiple data items stored in the database. A processing module is used to input the plurality of data into a sorting model to obtain a score corresponding to each of the data, wherein the input information of the sorting model includes the plurality of data, the output information of the sorting model includes the score corresponding to each of the data, and the sorting model is determined based on the plurality of data and at least one query command, wherein the query command is used to query target data from the plurality of data. The processing module is further configured to determine the file blocks corresponding to each of the multiple data based on the score corresponding to each of the data.

9. The apparatus according to claim 8, characterized in that, The device also includes a training module. The acquisition module is further configured to acquire the at least one query command and sample data, wherein the sample data is obtained by sampling the stored plurality of data; The training module is used to train the ranking model based on the at least one query command and the sample data.

10. The apparatus according to claim 8 or 9, characterized in that, The device further includes: A compression module is used to compress the multiple data to obtain corresponding multiple dimensionality-reduced data; The processing module is specifically used to input the multiple dimensionality-reduced data into the sorting model to obtain a score corresponding to each data point.

11. The apparatus according to claim 10, characterized in that, The compression module is specifically used for: The multiple data are input into a compression model to obtain the multiple dimensionality-reduced data, wherein the input information of the compression model includes the multiple data, and the output information of the compression model includes the multiple dimensionality-reduced data.

12. The apparatus according to claim 11, characterized in that, The training module is also used to train the compressed model based on the sample data.

13. The apparatus according to claim 11 or 2, characterized in that, The compression model is a variational autoencoder (VAE).

14. The method according to any one of claims 8 to 13, characterized in that, The device is applied to a cloud management platform, which manages the infrastructure that provides cloud services. The infrastructure includes at least one cloud data center, and each cloud data center is equipped with at least one server, which is used to train a ranking model and a compression model.

15. A computing device cluster, characterized in that, It includes at least one computing device, each computing device including a processor and memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the cluster of computing devices to perform the method as described in any one of claims 1 to 7.

16. A computer program product containing instructions, characterized in that, When the instruction is executed by the computing device cluster, the computing device cluster performs the method as described in any one of claims 1 to 7.

17. A computer-readable storage medium, characterized in that, It includes computer program instructions, which, when executed by a cluster of computing devices, perform the method as described in any one of claims 1 to 7.

Images