A method, apparatus, device, and storage medium for training a data classification model.

Abstract

This application provides a method, apparatus, device, and storage medium for training data classification models, applicable to fields such as artificial intelligence and big data, addressing the problem of low efficiency in training data classification models. The method includes: controlling multiple storage nodes, based on received data selection instructions, to select multiple candidate data as multiple training data from various stored candidate data; controlling the multiple storage nodes, based on the multiple training data, to perform multiple rounds of iterative training on the corresponding data classification model to be trained, obtaining multiple trained data classification models; controlling a computing node to retrieve the multiple trained data classification models from the multiple storage nodes, and performing model fusion on the multiple trained data classification models to obtain a target data classification model.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method, apparatus, device, and storage medium for training a data classification model. Background Technology

[0002] With the continuous development of technology, more and more devices can provide data classification services through trained data classification models. Data classification services can be used to determine the category to which data belongs.

[0003] In related technologies, the method for obtaining a trained data classification model typically involves the device first building the data classification model to be trained locally, then sending a data retrieval request to the storage unit, which returns the training data to the device based on the request. Based on the obtained training data, the device performs multiple rounds of iterative training on the data classification model locally, thus obtaining the trained data classification model. Therefore, upon obtaining the trained data classification model, the device can read the stored data from the storage unit and sequentially classify the stored data using the data classification model.

[0004] However, the device needs to send a data retrieval request to the storage unit and wait for the storage unit to respond to the data retrieval request before it can obtain each training data. This makes the process of obtaining training data relatively complicated, resulting in the problem of low efficiency of the training data classification model.

[0005] It is evident that among related technologies, the efficiency of training data classification models is relatively low. Summary of the Invention

[0006] This application provides a method, apparatus, computer device, and storage medium for training a data classification model, which addresses the problem of low efficiency in training data classification models.

[0007] Firstly, a method for training a data classification model is provided, applied to a distributed data space, wherein the distributed data space includes computing nodes and multiple storage nodes, the computing nodes are used to process data, and the storage nodes are used to store data, including:

[0008] Based on the received data selection instructions, the multiple storage nodes are controlled to select multiple candidate data from the stored candidate data as multiple training data.

[0009] The multiple storage nodes are controlled respectively, and the corresponding data classification models to be trained are iteratively trained in multiple rounds based on the multiple training data to obtain multiple trained data classification models.

[0010] The computing node is controlled to obtain the multiple trained data classification models from the multiple storage nodes, and the multiple trained data classification models are fused to obtain the target data classification model.

[0011] Optionally, before controlling the plurality of storage nodes respectively to select multiple candidate data from the stored candidate data as multiple training data, the method further includes:

[0012] When a model creation instruction is received from a client, a communication connection is established between the client and the computing node, wherein the model creation instruction carries initial model parameters;

[0013] Based on the initial model parameters, a classification model for the data to be trained is built on the computing node;

[0014] The computing node is controlled to distribute the classification model of the data to be trained to the multiple storage nodes.

[0015] Optionally, controlling the computing nodes to distribute the training data classification model to the multiple storage nodes includes:

[0016] Control the computing node to store the classification model of the data to be trained into the cache;

[0017] Receive model training instructions from the client and control the computing node to forward the model training instructions to the multiple storage nodes;

[0018] Each of the multiple storage nodes is controlled to read the classification model of the data to be trained from the cache when it receives the model training instruction.

[0019] Optionally, the multiple storage nodes are controlled separately to perform multiple rounds of iterative training on the corresponding data classification model to be trained based on multiple selected training data, thereby obtaining multiple trained data classification models, including:

[0020] During each round of training iterations, perform the following operations:

[0021] Each of the multiple storage nodes is controlled to perform an iterative training on the corresponding classification model of the data to be trained based on a training data, thereby obtaining the corresponding training loss and calculating the corresponding training time.

[0022] The multiple storage nodes are controlled respectively. When it is determined that the obtained training loss does not meet the corresponding training objective, or when it is determined that the statistical training time exceeds the corresponding preset time, the next round of iterative training is entered.

[0023] By controlling the multiple storage nodes respectively, and determining when the obtained training loss satisfies the corresponding training objective, multiple trained data classification models are obtained.

[0024] Optionally, controlling the computing node to obtain the multiple trained data classification models from the multiple storage nodes, and performing model fusion on the multiple trained data classification models to obtain the target data classification model, includes:

[0025] Control the computing node to obtain the multiple trained data classification models from the multiple storage nodes;

[0026] The computing node is controlled to perform weighted fusion of the training model parameters of each trained data classification model to obtain comprehensive model parameters;

[0027] The computing node is controlled to build the target data classification model based on the comprehensive model parameters and the model structure of the trained data classification model.

[0028] Optionally, the computing node is configured to build the target data classification model based on the comprehensive model parameters and the model structure of the trained data classification model, including:

[0029] Control the computing node to build a data classification model to be verified based on the comprehensive model parameters and the model structure of the trained data classification model;

[0030] The multiple storage nodes are controlled respectively to select multiple candidate data from the stored candidate data as verification data, wherein the verification data has associated classification labels;

[0031] The computing node is controlled to receive multiple verification data from the multiple storage nodes, and the data to be verified classification model is used to determine the predicted classification of each of the multiple verification data.

[0032] The computing node is controlled to use the data classification model to be verified as the target data classification model when the error between the obtained predicted classification and the corresponding classification label meets the preset error condition.

[0033] Secondly, an apparatus for training a data classification model is provided, applied to a distributed data space, the distributed data space including computing nodes and multiple storage nodes, the computing nodes being used to process data, and the storage nodes being used to store data, including:

[0034] Acquisition module: Based on the received data selection instructions, it controls the multiple storage nodes to select multiple candidate data from the stored candidate data as multiple training data;

[0035] Processing module: Used to control the multiple storage nodes respectively, and perform multiple rounds of iterative training on the corresponding data classification model to be trained based on the multiple training data to obtain multiple trained data classification models;

[0036] The processing module is further configured to: control the computing node to obtain the multiple trained data classification models from the multiple storage nodes, and perform model fusion on the multiple trained data classification models to obtain the target data classification model.

[0037] Optionally, the processing module is further configured to:

[0038] Before controlling the multiple storage nodes to select multiple candidate data as multiple training data from the stored candidate data, when receiving a model creation instruction from the client, a communication connection is established between the client and the computing node, wherein the model creation instruction carries initial model parameters;

[0039] Based on the initial model parameters, a classification model for the data to be trained is built on the computing node;

[0040] The computing node is controlled to distribute the classification model of the data to be trained to the multiple storage nodes.

[0041] Optionally, the processing module is specifically used for:

[0042] Control the computing node to store the classification model of the data to be trained into the cache;

[0043] Receive model training instructions from the client and control the computing node to forward the model training instructions to the multiple storage nodes;

[0044] Each of the multiple storage nodes is controlled to read the classification model of the data to be trained from the cache when it receives the model training instruction.

[0045] Optionally, the processing module is specifically used for:

[0046] During each round of training iterations, perform the following operations:

[0047] Each of the multiple storage nodes is controlled to perform an iterative training on the corresponding classification model of the data to be trained based on a training data, thereby obtaining the corresponding training loss and calculating the corresponding training time.

[0048] The multiple storage nodes are controlled respectively. When it is determined that the obtained training loss does not meet the corresponding training objective, or when it is determined that the statistical training time exceeds the corresponding preset time, the next round of iterative training is entered.

[0049] By controlling the multiple storage nodes respectively, and determining when the obtained training loss satisfies the corresponding training objective, multiple trained data classification models are obtained.

[0050] Optionally, the processing module is specifically used for:

[0051] Control the computing node to obtain the multiple trained data classification models from the multiple storage nodes;

[0052] The computing node is controlled to perform weighted fusion of the training model parameters of each trained data classification model to obtain comprehensive model parameters;

[0053] The computing node is controlled to build the target data classification model based on the comprehensive model parameters and the model structure of the trained data classification model.

[0054] Optionally, the processing module is specifically used for:

[0055] Control the computing node to build a data classification model to be verified based on the comprehensive model parameters and the model structure of the trained data classification model;

[0056] The multiple storage nodes are controlled respectively to select multiple candidate data from the stored candidate data as verification data, wherein the verification data has associated classification labels;

[0057] The computing node is controlled to receive multiple verification data from the multiple storage nodes, and the data to be verified classification model is used to determine the predicted classification of each of the multiple verification data.

[0058] The computing node is controlled to use the data classification model to be verified as the target data classification model when the error between the obtained predicted classification and the corresponding classification label meets the preset error condition.

[0059] Thirdly, a computer program product is provided, including a computer program that, when executed by a processor, implements the method described in the first aspect.

[0060] Fourthly, a computer device is provided, comprising:

[0061] Memory, used to store program instructions;

[0062] A processor is configured to invoke program instructions stored in the memory and execute the method described in the first aspect according to the obtained program instructions.

[0063] Fifthly, a computer-readable storage medium is provided, the computer-readable storage medium storing computer-executable instructions for causing a computer to perform the method as described in the first aspect.

[0064] In this embodiment, a distributed data space is combined with a data classification model. The data classification model is trained directly on multiple storage nodes contained in the distributed data space. This eliminates the need for computing nodes to obtain training data from storage nodes and for storage nodes to send training data to computing nodes, thereby improving the efficiency of training the data classification model.

[0065] Furthermore, each of the multiple storage nodes trains a classification model for the data to be trained. The compute nodes then fuse the trained classification models from the multiple storage nodes to obtain the target data classification model. Multiple storage nodes can achieve multiple rounds of parallel training, further improving the efficiency of training the classification model for the data to be trained. Attached Figure Description

[0066] Figure 1 This is one application scenario of the method for training data classification models provided in the embodiments of this application;

[0067] Figure 2 A flowchart illustrating a method for training a data classification model provided in an embodiment of this application;

[0068] Figure 3 An interactive illustration of the method for training data classification models provided in the embodiments of this application. Figure 1 ;

[0069] Figure 4 An interactive illustration of the method for training data classification models provided in the embodiments of this application. Figure 2 ;

[0070] Figure 5 A schematic diagram of the principle of the method for training data classification model provided in the embodiments of this application. Figure 1 ;

[0071] Figure 6 A schematic diagram of the principle of the method for training data classification model provided in the embodiments of this application. Figure 2 ;

[0072] Figure 7 A schematic diagram of the principle of the method for training data classification model provided in the embodiments of this application. Figure 3 ;

[0073] Figure 8 A schematic diagram of the apparatus for training data classification models provided in the embodiments of this application. Figure 1 ;

[0074] Figure 9 A schematic diagram of the apparatus for training data classification models provided in the embodiments of this application. Figure 2 . Detailed Implementation

[0075] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings.

[0076] It should be noted that in the embodiments of this application, data such as training data are involved. When the above embodiments of this application are applied to specific products or technologies, user permission or consent is required each time data is acquired, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0077] The following is a brief introduction to the application areas of the training data classification model method provided in the embodiments of this application.

[0078] With the continuous development of technology, more and more devices can provide data classification services through trained data classification models. Data classification services can be used to determine the category to which data belongs.

[0079] In related technologies, the method for obtaining a trained data classification model typically involves the device first building the data classification model to be trained locally, then sending a data retrieval request to the storage unit, which returns the training data to the device based on the request. Based on the obtained training data, the device performs multiple rounds of iterative training on the data classification model locally, thus obtaining the trained data classification model. Therefore, upon obtaining the trained data classification model, the device can read the stored data from the storage unit and sequentially classify the stored data using the data classification model.

[0080] For example, the advent of the big data era has made devices increasingly reliant on high-performance services and massive storage space. At the same time, more and more devices need databases that include both traditional transaction functions and analytical processes, leading to the increasing use of hybrid databases (Hybrid Transactional / Analytical Processing, HTAP). Since hybrid databases support both transactional and analytical processing, they typically require higher cluster sizes and machine specifications. Therefore, traditional database architectures can no longer meet the requirements for training data classification models, and combining big data development with distributed hybrid databases is gradually becoming a trend.

[0081] However, the device needs to send a data retrieval request to the storage unit and wait for the storage unit to respond to the data retrieval request before it can obtain each training data. This makes the process of obtaining training data relatively complicated, resulting in the problem of low efficiency of the training data classification model.

[0082] For example, some solutions that utilize the distributed characteristics of hybrid databases to distribute data storage only alleviate the storage and query pressure on a single machine. From the perspective of training data classification models, the problem of low training efficiency still exists.

[0083] It is evident that among related technologies, the efficiency of training data classification models is relatively low.

[0084] To address the issue of low efficiency in training data classification models, this application proposes a method for data classification models. This method, based on received data selection instructions, controls multiple storage nodes to select multiple candidate data points from a pool of stored data as training data. The multiple storage nodes are then controlled to iteratively train the corresponding data classification model on the target data using the multiple training data points, resulting in multiple trained data classification models. Finally, a computing node retrieves these trained data classification models from the storage nodes and fuses them to obtain the target data classification model.

[0085] In this embodiment, a distributed data space is combined with a data classification model. The data classification model is trained directly on multiple storage nodes contained in the distributed data space. This eliminates the need for computing nodes to obtain training data from storage nodes and for storage nodes to send training data to computing nodes, thereby improving the efficiency of training the data classification model.

[0086] Furthermore, each of the multiple storage nodes trains a classification model for the data to be trained. The compute nodes then fuse the trained classification models from the multiple storage nodes to obtain the target data classification model. Multiple storage nodes can achieve multiple rounds of parallel training, further improving the efficiency of training the classification model for the data to be trained.

[0087] The following describes the application scenarios of the training data classification model method provided in this application.

[0088] Please refer to Figure 1This diagram illustrates an application scenario of the training data classification model method provided in this application. The application scenario includes a client 101 and a server 102. The client 101 and the server 102 can communicate with each other. The communication method can be wired, such as through a network cable or serial cable; or wireless, such as through Bluetooth or Wi-Fi. No specific limitation is imposed.

[0089] Client 101 generally refers to a device that can send instructions to server 102, such as a terminal device, a third-party application accessible by the terminal device, or a webpage accessible by the terminal device. Terminal devices include, but are not limited to, mobile phones, computers, intelligent transportation equipment, and smart appliances. Server 102 generally refers to a device that can train and use a data classification model, such as a terminal device or a server. Servers include, but are not limited to, cloud servers, local servers, or associated third-party servers. Both client 101 and server 102 can use cloud computing to reduce the consumption of local computing resources; similarly, they can also use cloud storage to reduce the consumption of local storage resources.

[0090] As one embodiment, the client 101 and the server 102 can be the same device. For example, this embodiment can be implemented using only the client 101 or only the server 102, etc., without any specific limitation. In this embodiment, the client 101 and the server 102 are described as different devices.

[0091] The following is based on Figure 1 Taking server 102 as an example, the method for training data classification models provided in this application embodiment will be described in detail. This method is applied to a distributed data space, which includes computing nodes and multiple storage nodes. The computing nodes are used to process data, and the storage nodes are used to store data.

[0092] Please refer to Figure 2 This is a flowchart illustrating a method for training data classification models provided in an embodiment of this application.

[0093] S201, based on the received data selection instruction, controls multiple storage nodes to select multiple candidate data from the stored candidate data as multiple training data.

[0094] The client can send data selection instructions to the server. The server receives these instructions and, based on them, controls multiple storage nodes to select multiple candidate data points as training data from a pool of stored data. The data selection instructions can be SQL statements specifying the number of storage nodes from which to select training data.

[0095] Multiple storage nodes store various alternative data. For an alternative data stored on one storage node, a backup of that alternative data can be stored on other storage nodes. The backup data can be used as alternative data stored on other storage nodes, thereby avoiding situations where alternative data is lost or cannot be read for a long time due to read latency on a certain storage node.

[0096] Multiple candidate data points are selected from the stored candidate data points as multiple training data points. This can be done by selecting multiple candidate data points stored on one storage node from the various candidate data points, or by selecting multiple candidate data points stored on multiple storage nodes from the various candidate data points, etc. There are no specific restrictions.

[0097] As one implementation, before acquiring multiple training data sets, the client can send a model creation command to the server, which then receives the command. The model creation command can instruct the server to create a classification model for the training data with a specific model structure. For example, the server may store classification models for the training data with various structured models and different model names. The client can then instruct the server to create a classification model for the training data with a specific model name using the model creation command. Alternatively, the model creation command can carry the model structure of the classification model for the training data, allowing the server to establish a classification model for the training data with the corresponding structure.

[0098] When the server receives a model creation command from the client, it establishes a communication connection between the client and the computing node, allowing the client to interact with the server area for data exchange.

[0099] The model creation command can also carry initial model parameters. Based on these initial parameters, the server can build a classification model for the training data on the computing nodes according to the model structure specified in the model creation command. Alternatively, the server can build a classification model for the training data on the computing nodes based on the initial parameters and using the default model structure, etc., without any specific restrictions.

[0100] For example, a client can establish a communication connection with the compute nodes contained in the server. The client can send SQL statements, i.e., send model creation instructions, to the compute nodes through the SQL interface. For instance, the SQL statement could be: `CREATE MODEL nameWITH 'type' = 'NLP', 't1' = 'v1', 't2' = 'v2'`. This SQL statement creates a data classification model structure named "name" and initializes the initial model parameters, setting the initial model parameter "t1" to "v1" and the initial model parameter "t2" to "v2".

[0101] The server can store the model structure and initial model parameters of the data classification model in the distributed data space, and use the sub-controls to store model-related data. For example, when the distributed data space is a hybrid database, the name of the data classification model and the initial model parameters can be stored in the model-federated-learning library in the hybrid database. Create a table named "name" in the model-federated-learning library, set two fields named "t1" and "t2", and assign corresponding values ​​to them.

[0102] The server can continue to send SQL statements to the compute nodes via the SQL interface. The compute nodes then forward these SQL statements to multiple storage nodes, controlling them to select multiple candidate data points from their stored pool of data to serve as training data. For example, the SQL statement could be: `TRAIN MODEL name WITH 'select A'`. This SQL statement can function as both a data selection instruction and a model training instruction, directing multiple storage nodes to select data A to train a model named "name".

[0103] After obtaining the classification model for the training data, the server can control the computing nodes to distribute the model to multiple storage nodes. These storage nodes then train the model, eliminating the need for data transfer between computing and storage nodes and reducing unnecessary network interactions. Computing nodes can also distribute the model to multiple storage nodes via a parallel computing interface. Since the model structure and initial parameters are already stored in the distributed data space, computing nodes only need to send the model name to the storage nodes through the parallel computing interface, without transmitting additional data and further reducing network interactions.

[0104] As one embodiment, the process of controlling the computing node to distribute the training data classification model to multiple storage nodes can be achieved by first controlling the computing node to store the training data classification model in a cache after it has been created. When the computing node receives a model training instruction from the client, it forwards the instruction to multiple storage nodes. Thus, when multiple storage nodes receive the training instruction, they can read the training data classification model from the cache, achieving the goal of independently controlling multiple storage nodes to read the model from the cache when receiving the instruction. Therefore, the computing node does not need to distribute the training data classification model to the storage nodes in real time upon obtaining it; instead, it can store it in the cache first, and retrieve it from the cache when the storage nodes need to use it, improving training flexibility.

[0105] S202 controls multiple storage nodes respectively, and performs multiple rounds of iterative training on the corresponding data classification model to be trained based on multiple training data to obtain multiple trained data classification models.

[0106] After selecting multiple training data sets, the server can control multiple storage nodes to perform multiple rounds of iterative training on the corresponding classification models for the data to be trained, thereby obtaining multiple trained classification models. Each storage node corresponds to one classification model for the data to be trained, enabling multiple storage nodes to perform multiple rounds of parallel training, thus improving training efficiency.

[0107] The following example uses the process of one round of iterative training. The process of each round of iterative training is similar and will not be described in detail here.

[0108] During an iterative training round, the server can control multiple storage nodes to perform one round of training on the corresponding classification model based on a single training dataset, obtaining the corresponding training loss and recording the training duration. Each storage node corresponds to a classification model, and each classification model corresponds to a training loss and a training duration. The server can start timing when a storage node begins training the classification model based on a single training dataset and stop timing when the training loss is obtained.

[0109] The server can control multiple storage nodes separately to determine whether the obtained training loss meets the corresponding training objective. If the obtained training loss does not meet the corresponding training objective, the server adjusts the model parameters of the corresponding classification model for the data to be trained and enters the next round of iterative training.

[0110] If the training time exceeds the preset time and the corresponding training loss has not been obtained, then the training loss can be stopped, the model parameters of the classification model for the data to be trained can be adjusted directly, and the next round of iterative training can be started. This avoids situations where a single training round takes too long or training anomalies occur during a single training round, which could prevent the entire multi-round iterative training process from progressing.

[0111] Once it is determined that the obtained training loss satisfies the corresponding training objective, multiple trained data classification models are obtained. The server can control each storage node to obtain a trained data classification model when it determines that the obtained training loss satisfies the corresponding training objective, until all storage nodes have obtained the corresponding trained data classification models.

[0112] For example, each storage node first selects training data from the candidate data stored locally according to the instructions of the SQL statement, and then performs multiple rounds of iterative training on the classification model of the data to be trained under the initial model parameters indicated by the SQL statement based on the training data. After multiple rounds of iterative training, each storage node can obtain the training model parameters trained by that storage node, that is, obtain the trained data classification model under the training model parameters.

[0113] Each training iteration has a time limit, i.e., a preset duration. The computing node cannot wait indefinitely for the training results from the storage node. Therefore, the storage node ends its current training iteration not only when the training loss converges, but also when the training time exceeds the preset duration set by the computing node. Once the storage node has completed training using the training data stored locally, it obtains the trained data classification model with the trained model parameters.

[0114] S203 controls the computing node to obtain multiple trained data classification models from multiple storage nodes, and performs model fusion on the multiple trained data classification models to obtain the target data classification model.

[0115] After multiple storage nodes have obtained trained data classification models, the server can control the compute nodes to retrieve multiple trained data classification models from the storage nodes and fuse these models to obtain the target data classification model. Since the training performance of the data classification models on each storage node differs, the resulting trained data classification models will also differ. The server can control the compute nodes to fuse these multiple trained data classification models to obtain a unified target data classification model.

[0116] As one example, if the client specifies the model structure of the data classification model via a model creation command, then when the server controls the computing nodes to retrieve multiple trained data classification models from multiple storage nodes, it can control the computing nodes to obtain the training model parameters of these multiple trained data classification models from the multiple storage nodes. The server can then perform weighted fusion of the obtained training model parameters to obtain comprehensive model parameters. Thus, the server can control the computing nodes to build the target data classification model based on the comprehensive model parameters and the model structure specified in the model creation command.

[0117] As one embodiment, when obtaining a target data classification model based on comprehensive model parameters, the server can first control the computing nodes to build a data classification model to be verified based on the comprehensive model parameters and the model structure of the trained data classification model. Multiple storage nodes are controlled to select multiple candidate data from the stored candidate data as multiple verification data, where the verification data have associated classification labels. The computing nodes receive the multiple verification data from the multiple storage nodes and use the data classification model to be verified to determine the predicted classification of each of the multiple verification data. When the control node determines that the error between the obtained predicted classification and the corresponding classification label meets a preset error condition, it uses the data classification model to be verified as the target data classification model. After fusing the multiple trained data classification models, the accuracy of the fused model can be verified. If the accuracy meets the requirements, the target data classification model is obtained; if the accuracy does not meet the requirements, multiple rounds of iterative training are continued on the multiple trained data classification models until the accuracy of the fused model meets the requirements, thus avoiding the situation where the classification performance of the multiple trained data classification models after model fusion is poor.

[0118] For example, after obtaining the trained data classification model with the training model parameters, the storage node can send these parameters to the compute node. Upon receiving the training model parameters from the storage node, the compute node can use the FedAvg algorithm to fuse the training model parameters from each storage node and use data stored in a distributed data space, such as a distributed hybrid database, to verify the model's accuracy. If the accuracy does not meet the requirements, the next round of federated learning training is initiated. When the accuracy meets the requirements, the resulting global model is written to the hybrid database, and the training results are returned to the user.

[0119] The compute nodes can be considered as master nodes, and the multiple storage nodes can be considered as slave nodes. The master node is responsible for issuing training instructions to the model and fusing the training model parameters to obtain the target data classification model. The master node can also determine whether global convergence has been reached to end the current learning cycle. The slave nodes are responsible for training the model using the initial model data and providing feedback on the trained model parameters after iterative training.

[0120] As one embodiment, after obtaining the target data classification model, the computing node can use the target data classification model to provide data classification services. After receiving a data query retrieval instruction from the client, based on the keywords and retrieval scope contained in the query retrieval instruction, the computing node can obtain candidate data that meets the retrieval scope from each storage node, and then use the target data classification model to filter the candidate data that meets the retrieval scope to include the target data containing the keywords.

[0121] For example, after obtaining the comprehensive model parameters, the compute node can use the built-in function APPLY to apply the target data classification model. The client sends an SQL statement to the compute node via the SQL interface. The SQL statement is: SELECT * FROM image WHERE APPLY('image-model' = 'car'). This SQL statement is used to apply the target data classification model named 'image-model' and search for target data containing the keyword 'car' in the table 'image'.

[0122] In this embodiment, compared to existing solutions combining big data and hybrid databases, the model training for big data is introduced into a hybrid database cluster, fully leveraging the high performance and distributed storage advantages of the hybrid database cluster. To simultaneously support transactional and analytical queries, hybrid databases typically use large-scale clusters with high-specification machines, providing a foundation for distributed storage and model training of big data. By distributing data samples across multiple nodes, distributed training is achieved across multiple nodes, and then a single node aggregates the parameters to obtain a final model.

[0123] Furthermore, in this embodiment, a corresponding SQL interface for selecting training data and training models is designed, enabling users to more conveniently and directly use the hybrid database to complete big data development through SQL statements.

[0124] In this embodiment, the massive storage requirements of big data can be met, and high availability is achieved through a multi-replica inter-node data scheduling scheme, fully leveraging the distributed characteristics and high-performance advantages of hybrid databases. Users can train and use the model through the SQL interface and built-in functions, reducing the learning cost for users and making it more convenient and direct to operate the federated learning big data system based on hybrid databases, which is conducive to the promotion and implementation of the product.

[0125] Please refer to Figure 3 This is a schematic diagram illustrating the interaction between the client, computing nodes, and storage nodes when training a classification model for the data to be trained. The example is a storage node.

[0126] S301, the client sends a model creation command to the computing nodes contained in the distributed data space of the server. The model creation command carries the name of the model structure and the initial model parameters. The computing nodes receive the model creation command sent by the client and obtain the name of the model structure and the initial model parameters.

[0127] S302, the computing node builds a classification model for the data to be trained based on the model creation instruction, with the corresponding model structure and initial model parameters.

[0128] S303, the compute node sends the data classification model to be trained to the storage node. The compute node can first store the name of the model structure and the initial training parameters of the data classification model to be trained, and then send the name of the model structure and the initial training parameters of the data classification model to be trained to the storage node. The storage node receives the name of the model structure and the initial training parameters of the data classification model to be trained sent by the compute node, and then sends the data classification model to be trained.

[0129] S304, the client sends a data selection instruction to the compute node. This instruction can be used to instruct the selection of multiple candidate data sets from the storage node as training data. The compute node receives the data selection instruction from the client and forwards it to the storage node. The storage node then receives the forwarded data selection instruction from the compute node.

[0130] S305, the storage node selects multiple candidate data as multiple training data based on the data selection instruction.

[0131] S306, the storage node performs multiple rounds of iterative training on the corresponding data classification model to be trained based on multiple training data to obtain the trained data classification model.

[0132] S307, after obtaining the trained data classification model, the storage node sends the trained data classification model to the compute node. The storage node may also send only the training model parameters of the trained data classification model to the compute node. The compute node receives the trained data classification model, i.e., the training model parameters, sent by the storage node.

[0133] S308: After obtaining the trained data classification models sent by each storage node, the compute node performs model fusion on the multiple trained data classification models to obtain the target data classification model. After obtaining the training model parameters sent by each storage node, the compute node can perform weighted fusion on the obtained training model parameters to obtain comprehensive model parameters. Based on the comprehensive model parameters and the model structure of the data classification model to be trained, the target data classification model is obtained.

[0134] Please refer to Figure 4 This is a schematic diagram illustrating the interaction between the client, computing node, and storage node when using a classification model for the data to be trained. It is illustrated using a storage node as an example.

[0135] In step S401, the client sends a data retrieval command to the computing nodes within the distributed data space on the server. This command is used to search for target data among candidate data stored on various storage nodes. The data retrieval command may include the target data classification model, the search scope, and key information.

[0136] S402, the compute node retrieves candidate data within the retrieval range from multiple storage nodes based on data retrieval instructions. The storage nodes send the candidate data within the retrieval range stored locally to the compute node.

[0137] S403, the computing node uses a target data classification model to search for target data. The computing node uses the target data classification model to search for target data containing key information among the candidate data within the retrieval range.

[0138] The following provides examples of the training data classification model and the method for using the trained target data classification model provided in the embodiments of this application. Please refer to... Figure 5 This is a schematic diagram of a structure. Clients can interact with the server, and the server can select at least one computing node from multiple computing nodes for data processing through load balancing. Computing nodes are stateless nodes and are not used for data storage; adding computing nodes results in a linear performance improvement.

[0139] At least one compute node can interact with the storage node cluster. To support both transactional and analytical queries, the storage nodes are divided into row-based and column-based nodes. After data sharding, it is stored as multiple replicas across multiple storage nodes, achieving high availability and parallel querying. This data sharding scheme meets the massive storage needs of big data development and improves query efficiency through parallel queries.

[0140] Row-based storage nodes use the same storage method as traditional databases, storing user data row-by-row in the storage medium. Column-based storage nodes, on the other hand, break down each field in a row, storing the data as a column for each field. Traditional transactional queries require reading the entire dataset; row-based storage nodes can handle this distributed querying. Analytical transactional queries typically only need to read certain fields from a row; column-based storage nodes can reduce disk overhead and thus query time by reading columns separately. The choice between row-based and column-based storage nodes is determined by the compute node based on the user's SQL statement. Providing both row-based and column-based storage facilitates data reading and analysis for big data development.

[0141] Please refer to Figure 6 In addition to compute nodes and storage nodes, a management node may also be included. The role of the management node is to coordinate the replicas on various storage nodes. When hot data appears on a storage node, the management node needs to promptly distribute the replica to multiple other storage nodes to avoid putting pressure on the hybrid database. Furthermore, the management node stores the storage information of all replicas. When the cached replica storage information in the compute nodes expires, a request needs to be sent to the management node.

[0142] Please refer to Figure 7 This is a schematic diagram illustrating the principle of a training data classification model provided in this application embodiment, and a method for using the trained target data classification model.

[0143] In response to the first input operation triggered by the user, the client sends a model creation command to the computing node in the server. The first input operation is to input an SQL statement, which is "CREATE MODEL name WITH 'type' = 'GAN', 'k1' = 'v1', 'k2' = 'v2'". This SQL statement is used to create a model structure named 'name' with type 'GAN' on the computing node, and the model parameters of the training data classification model are 'k1' = 'v1' and 'k2' = 'v2'.

[0144] In response to a second input operation triggered by the user, a data selection instruction is sent to the compute node. The second input operation is to input an SQL statement, which is "TRAIN MODEL name WITH 'select A'". This SQL statement can be used to control the compute node to distribute the classification model of the data to be trained to multiple storage nodes, and it can also be used to select multiple training data on the storage nodes.

[0145] Each storage node obtains the classification model for the data to be trained and selects multiple training data sets before initiating the first round of iterative training. Each storage node processes the stored data according to the user's selection statement and performs iterative training. Upon convergence or timeout, the trained model parameters are returned to the compute node that initiated the distributed training. The compute node uses the FedAvg algorithm to fuse the training model parameters returned by each storage node to obtain comprehensive model parameters. It then determines whether the data classification model under the obtained comprehensive model parameters has reached the convergence target. If not, the next round of iterative training begins. Once global convergence is achieved, the iterative training ends, and the target data classification model is obtained.

[0146] After obtaining the target data classification model, in response to a third input operation triggered by the user, a data retrieval instruction is sent to the computing node. The computing node then uses the target data classification model through the built-in function "APPLY". The third input operation can be an SQL statement, such as "SELECT * FROM XXX where apply('name', YYY.img) = 'ZZZ'". This SQL statement is used to search for target data containing the image YYY.img in table XXX using the target data classification model named 'name'.

[0147] Based on the same inventive concept, embodiments of this application provide an apparatus for training a data classification model, capable of implementing the functions corresponding to the aforementioned method for training a data classification model. Please refer to... Figure 6 The device includes an acquisition module 801 and a processing module 802, wherein:

[0148] Acquisition module 801: Based on the received data selection instructions, it controls multiple storage nodes to select multiple candidate data from the stored candidate data as multiple training data;

[0149] Processing module 802: Used to control multiple storage nodes respectively, and perform multiple rounds of iterative training on the corresponding data classification model to be trained based on multiple training data to obtain multiple trained data classification models;

[0150] The processing module 802 is also used to: control the computing node to obtain multiple trained data classification models from multiple storage nodes, and perform model fusion on the multiple trained data classification models to obtain the target data classification model.

[0151] In one possible embodiment, the processing module 802 is further configured to:

[0152] Before controlling multiple storage nodes separately and selecting multiple candidate data from the stored candidate data as multiple training data, when receiving the model creation command from the client, a communication connection is established between the client and the computing node, wherein the model creation command carries the initial model parameters.

[0153] Based on the initial model parameters, a classification model for the data to be trained is built on the computing nodes;

[0154] Control the computing nodes and distribute the classification model of the data to be trained to multiple storage nodes.

[0155] In one possible embodiment, the processing module 802 is specifically used for:

[0156] Control the computing nodes and store the classification model of the data to be trained into the cache;

[0157] It receives model training instructions from the client and controls the computing nodes to forward the model training instructions to multiple storage nodes;

[0158] Multiple storage nodes are controlled to read the classification model from the cache when a model training instruction is received.

[0159] In one possible embodiment, the processing module 802 is specifically used for:

[0160] During each round of training iterations, perform the following operations:

[0161] Multiple storage nodes are controlled to perform one round of iterative training on the corresponding classification model of the data to be trained based on a single training data, thereby obtaining the corresponding training loss and calculating the corresponding training time.

[0162] Multiple storage nodes are controlled separately. When it is determined that the obtained training loss does not meet the corresponding training objective, or when it is determined that the statistical training time exceeds the corresponding preset time, the next round of iterative training is initiated.

[0163] By controlling multiple storage nodes separately, and determining when the obtained training loss satisfies the corresponding training objective, multiple trained data classification models are obtained.

[0164] In one possible embodiment, the processing module 802 is specifically used for:

[0165] Control computing nodes to retrieve multiple trained data classification models from multiple storage nodes;

[0166] Control the computing nodes to perform weighted fusion of the training model parameters of each trained data classification model to obtain comprehensive model parameters;

[0167] Control the computing nodes and build a target data classification model based on the comprehensive model parameters and the model structure of the trained data classification model.

[0168] In one possible embodiment, the processing module 802 is specifically used for:

[0169] Control computing nodes and build a data classification model to be validated based on the comprehensive model parameters and the model structure of the trained data classification model;

[0170] Multiple storage nodes are controlled separately, and multiple candidate data are selected from the various candidate data stored as verification data. The verification data has associated classification labels.

[0171] The control computing node receives multiple verification data from multiple storage nodes and uses the data classification model to determine the predicted classification of each of the multiple verification data.

[0172] The control computing node determines that when the error between the obtained predicted classification and the corresponding classification label meets the preset error condition, the classification model of the data to be verified is used as the target data classification model.

[0173] Please refer to Figure 9 The aforementioned training data classification model apparatus can run on a computer device 900. The current and historical versions of the data storage program, as well as the application software corresponding to the data storage program, can be installed on the computer device 900, which includes a processor 980 and a memory 920. In some embodiments, the computer device 900 may include a display unit 940, which includes a display panel 941 for displaying a user-interactive interface, etc.

[0174] In one possible embodiment, the display panel 941 may be configured in the form of a liquid crystal display (LCD) or an organic light-emitting diode (OLED).

[0175] The processor 980 is used to read a computer program and then execute the methods defined by the computer program. For example, the processor 980 reads a data storage program or file, thereby running the data storage program on the computer device 900 and displaying the corresponding interface on the display unit 940. The processor 980 may include one or more general-purpose processors, and may also include one or more DSPs (Digital Signal Processors) for performing related operations to implement the technical solutions provided in the embodiments of this application.

[0176] The memory 920 generally includes main memory and secondary storage. Main memory can be random access memory (RAM), read-only memory (ROM), and cache, etc. Secondary storage can be a hard disk, optical disk, USB flash drive, floppy disk, or magnetic tape drive, etc. The memory 920 is used to store computer programs and other data. The computer programs include applications corresponding to each client, and other data may include data generated after the operating system or applications are run, including system data (e.g., operating system configuration parameters) and user data. In this embodiment, program instructions are stored in the memory 920, and the processor 980 executes the program instructions in the memory 920 to implement any of the methods described in the preceding figures.

[0177] The aforementioned display unit 940 is used to receive input digital information, character information, or contact touch operations / non-contact gestures, and to generate signal inputs related to user settings and function control of the computer device 900. Specifically, in this embodiment, the display unit 940 may include a display panel 941. The display panel 941, for example, is a touch screen, which can collect touch operations performed by the user on or near it (such as operations performed by the user using a finger, stylus, or any suitable object or accessory on or on the display panel 941), and drive corresponding connection devices according to a pre-set program.

[0178] In one possible embodiment, the display panel 941 may include two parts: a touch detection device and a touch controller. The touch detection device detects the player's touch position and the signal generated by the touch operation, and transmits the signal to the touch controller. The touch controller receives touch information from the touch detection device, converts it into touch point coordinates, and sends it to the processor 980. It can also receive and execute commands from the processor 980.

[0179] The display panel 941 can be implemented using various types such as resistive, capacitive, infrared, and surface acoustic wave. In addition to the display unit 940, in some embodiments, the computer device 900 may also include an input unit 930. The input unit 930 may include an image input device 931 and other input devices 932, wherein the other input devices may include, but are not limited to, one or more of the following: a physical keyboard, function keys (such as volume control buttons, power buttons, etc.), a trackball, a mouse, and a joystick.

[0180] In addition to the above, the computer device 900 may also include a power supply 990 for powering other modules, an audio circuit 960, a near-field communication module 970, and an RF circuit 910. The computer device 900 may also include one or more sensors 950, such as an accelerometer, a light sensor, and a pressure sensor. The audio circuit 960 specifically includes a speaker 961 and a microphone 962, for example, the computer device 900 can use the microphone 962 to collect the user's voice and perform corresponding operations.

[0181] As one embodiment, the number of processors 980 can be one or more, and the processors 980 and the memory 920 can be coupled together or relatively independent.

[0182] As one example, Figure 9 The processor 980 in the middle can be used to implement, for example Figure 8 The functions of the acquisition module 801 and the processing module 802 in the process.

[0183] As one example, Figure 9 The processor 980 in the text can be used to implement the functions of the server or terminal devices discussed above.

[0184] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0185] Alternatively, if the integrated units of this invention are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this invention, or the parts that contribute to the prior art, can be embodied in the form of software products, for example, through a computer program product. This computer program product is stored in a storage medium and includes several instructions to cause a computer device to execute all or part of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, RAM, magnetic disks, or optical disks.

[0186] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0187] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A method for training a data classification model, characterized in that, This is applied to a distributed data space, which includes computing nodes and multiple storage nodes. The computing nodes are used to process data, and the storage nodes are used to store data. Each storage node stores backup data in other storage nodes, including: Based on the SQL statement received by the computing node as a model creation instruction, the multiple storage nodes are controlled to build a classification model for the data to be trained based on the name of the model structure and the initial model parameters carried by the model creation instruction. Based on the SQL statement received by the computing node as a data selection instruction, the multiple storage nodes are controlled to execute: one storage node selects multiple candidate data from each candidate data stored in at least one of the multiple storage nodes as multiple training data. The multiple storage nodes are controlled separately, and based on the multiple training data, the corresponding data classification models to be trained are trained in multiple rounds of iterative training to obtain multiple trained data classification models; the data classification models to be trained corresponding to the multiple storage nodes have the same model structure. The computing node is controlled to obtain the training model parameters of the multiple trained data classification models from the multiple storage nodes, and the training model parameters are weighted and fused to obtain comprehensive model parameters; and the target data classification model is obtained based on the name of the model structure and the comprehensive model parameters.

2. The method according to claim 1, characterized in that, Before controlling the multiple storage nodes to select multiple candidate data from the stored candidate data as multiple training data, the process further includes: When a model creation instruction is received from a client, a communication connection is established between the client and the computing node, wherein the model creation instruction carries initial model parameters; Based on the initial model parameters, a classification model for the data to be trained is built on the computing node; The computing node is controlled to distribute the classification model of the data to be trained to the multiple storage nodes.

3. The method according to claim 2, characterized in that, Controlling the computing nodes to distribute the training data classification model to the multiple storage nodes includes: Control the computing node to store the classification model of the data to be trained into the cache; Receive model training instructions from the client and control the computing node to forward the model training instructions to the multiple storage nodes; Each of the multiple storage nodes is controlled to read the classification model of the data to be trained from the cache when it receives the model training instruction.

4. The method according to claim 1, characterized in that, The multiple storage nodes are controlled respectively, and based on multiple selected training data, the corresponding data classification models to be trained are iteratively trained in multiple rounds to obtain multiple trained data classification models, including: During each round of training iterations, perform the following operations: Each of the multiple storage nodes is controlled to perform an iterative training on the corresponding classification model of the data to be trained based on a training data, thereby obtaining the corresponding training loss and calculating the corresponding training time. The multiple storage nodes are controlled respectively. When it is determined that the obtained training loss does not meet the corresponding training objective, or when it is determined that the statistical training time exceeds the corresponding preset time, the next round of iterative training is entered. By controlling the multiple storage nodes respectively, and determining when the obtained training loss satisfies the corresponding training objective, multiple trained data classification models are obtained.

5. The method according to claim 1, characterized in that, Controlling the computing node to build the target data classification model based on the comprehensive model parameters and the model structure of the trained data classification model includes: Control the computing node to build a data classification model to be verified based on the comprehensive model parameters and the model structure of the trained data classification model; The multiple storage nodes are controlled respectively to select multiple candidate data from the stored candidate data as multiple verification data, wherein the verification data has associated classification labels; The computing node is controlled to receive the multiple verification data from the multiple storage nodes, and the data to be verified classification model is used to determine the predicted classification of each of the multiple verification data. The computing node is controlled to use the data classification model to be verified as the target data classification model when the error between the obtained predicted classification and the corresponding classification label meets the preset error condition.

6. An apparatus for training a data classification model, characterized in that, This is applied to a distributed data space, which includes computing nodes and multiple storage nodes. The computing nodes are used to process data, and the storage nodes are used to store data. Each storage node stores backup data in other storage nodes, including: The acquisition module is used to control the multiple storage nodes to build a classification model for the data to be trained based on the SQL statement received by the computing node as a model creation instruction, and to control the multiple storage nodes to execute the following based on the SQL statement received by the computing node as a data selection instruction: one storage node selects multiple candidate data as multiple training data from the candidate data stored in at least one of the multiple storage nodes; Processing module: Used to control the multiple storage nodes respectively, and perform multiple rounds of iterative training on the corresponding data classification models to be trained based on the multiple training data to obtain multiple trained data classification models; the data classification models to be trained corresponding to the multiple storage nodes have the same model structure; The processing module is further configured to: control the computing node to obtain the training model parameters of the multiple trained data classification models from the multiple storage nodes, and perform weighted fusion of each training model parameter to obtain comprehensive model parameters; and obtain the target data classification model based on the name of the model structure and the comprehensive model parameters.

7. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the method as described in any one of claims 1 to 5.

8. A computer device, characterized in that, include: Memory, used to store program instructions; A processor is configured to invoke program instructions stored in the memory and execute the method as described in any one of claims 1 to 5 according to the obtained program instructions.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions for causing a computer to perform the method as described in any one of claims 1 to 5.

Images