A database service platform based on a large AI model and its usage method

Abstract

The present invention provides a database service platform based on an AI large model and a usage method. Among them, the database service platform is divided into a central computing layer, a dynamic topology node layer, and a user layer. The non-autonomous user nodes in the user layer upload user data slices, and the nodes in the dynamic topology node layer and the autonomous nodes in the user layer send the model parameters obtained after training the sample data to the central computing layer; the central computing layer performs weighted average calculation on the model parameters to obtain the aggregated total model parameters. The above solution can fully train user samples, ensure user privacy and security, and improve the efficiency and accuracy of model training.

Description

Technical Field

[0001] The present invention relates to the technical field of AI large models, and more specifically, to a database service platform based on an AI large model and a usage method thereof. Background Art

[0002] In recent years, technologies related to the artificial intelligence (AI) industry have developed rapidly and are being applied more and more widely and deeply in various industries with completely different technical principles, such as healthcare, finance, industrial manufacturing, autonomous driving, and supply chain. The development of AI technology is based on data, which is the core resource supporting the development of AI technology. Generally, AI large model data has various characteristics such as large quantity, multi-modal, fluidity, complexity, etc. Importantly, data samples imply user privacy behaviors. For example, medical data contains users' medical privacy, code data contains the future decision-making characteristics of a company, and user behavior data contains users' daily life information, etc. These highly sensitive big data will introduce huge risks after being exposed in an AI large model. In federated learning technology, data is stored in an autonomous database, and users independently manage data assets. Multiple participants (such as devices, enterprises, or institutions) collaborate to train and share models on the premise of keeping data local, and the original data does not need to leave the local. Independently managing data assets can basically solve the problem of data privacy exposure, but this distributed architecture actually has many drawbacks. First, not every participant has the software and hardware environment basis for model deployment and training, and small and micro participants (compared with large participants) are difficult to integrate and have low utilization rates. Second, the maintenance and alignment of models and data require all participants to synchronize, which will introduce complex problems other than technology such as management and decision-making. Third, when synthesizing the training parameters obtained from the training models of different participants for an asynchronous shared model or sample, it will lead to problems such as invalid / unbalanced parameters and inaccurate models. Finally, it is difficult to prevent model distortion problems, such as any participant fabricating or forging big data samples, resulting in overall model distortion. To solve the above problems, existing federated learning technology has conducted various researches and improvements in ensuring the deployment environment, model synchronization, compensating for the differences in data imbalance, data calibration, participation permissions, etc.; including adopting a distributed / hierarchical storage architecture, and adopting complex database logics (such as distributed databases (HDFS architecture, Spark computing framework), time series databases (InfluxDB / TimescaleDB), vector databases (Pinecone embedded data representation / PostageSQL+pgvector architecture), multi-modal databases (MongoDB / Neo4j), blockchain databases, etc.). However, these AI data management methods have not changed the characteristics of autonomous data management itself, so the problem of federated learning model distortion caused by the above situation cannot be completely avoided.

[0003] The information disclosed in the background art section of the present invention is only intended to deepen the understanding of the general background art of the present invention, and should not be regarded as an admission or any form of implication that this information constitutes the prior art known to those skilled in the art. Summary of the Invention

[0004] The present invention provides a database service platform based on an AI large model and a usage method thereof, which can fully train user samples, ensure user privacy and security, and improve the efficiency and accuracy of model training.

[0005] In a first aspect, an embodiment of the present disclosure provides a method for using a database service based on an AI large model, including:

[0006] S100, dividing the database service platform into a central computing layer, a dynamic topology node layer, and a user layer; the database service platform receives user node registration information from the user layer and returns a user token and node set information of the dynamic topology node layer to the user node;

[0007] S110, a non-autonomous user node in the user layer generates a data upload mapping table based on the node set information and a random algorithm, the data upload mapping table includes user token random digest information, and the data upload mapping table is used to map nodes in the node set of the dynamic topology node layer where the user layer data is stored; the user layer uploads user data slices to the mapped nodes based on the data upload mapping table;

[0008] S120, the nodes in the dynamic topology node layer receive the user data slices, perform node standardization processing on the user data slices in the corresponding data format of the node, perform confusion processing on the data slices after the node standardization processing, and store them in the database sub-nodes; the training sub-nodes train the sample data in the database sub-nodes based on the configured AI model and send the trained model parameters to the central computing layer; the autonomous nodes in the user layer train their sample data based on the configured AI model and send the trained model parameters to the central computing layer;

[0009] S130, the central computing layer receives the model parameters of each node in the dynamic topology node layer and the model parameters of the autonomous nodes in the user layer, and performs weighted average calculation on the model parameters to obtain the aggregated total model parameters.

[0010] Preferably, the user slices the user data to be uploaded by unit. The slicing algorithm is a random generation algorithm, which ensures that the minimum slicing unit is the data unit that can be processed by training. The non-autonomous node generates a data upload mapping table based on the received information of the node set in the dynamic topology node layer and the random algorithm. The data upload mapping table is used to map the nodes in the node set of the dynamic topology node layer stored in the user layer data, adapt the user data slice and the corresponding upload path, and thus upload the data slice according to the upload path of the nodes in the mapped dynamic topology node layer node set.

[0011] Preferably, the implementation manner of the confusion processing is as follows: (1) temporarily store the received data slice in a non-decrypted state; (2) within a preset time, judge whether the data source of the data slice is less than a predetermined quantity threshold. If so, decrypt the data slice and add controllable noise data, and store it in the database sub-node; if not, perform confusion processing on the data in the data slices from different data sources and store it in the database sub-node.

[0012] Preferably, the dynamic topology node layer includes a specific number of distributed physical nodes. Within a preset time, judge whether the number of nodes whose data source of the data slice is less than the predetermined quantity threshold exceeds the second predetermined quantity threshold. If it exceeds, select several of them for shutdown processing. Before shutdown, dynamically migrate its data to other nodes and release the physical node; at the same time, within a preset time, judge whether the number of nodes whose data source of the data slice is greater than the third predetermined quantity threshold exceeds the fourth predetermined quantity threshold. If it exceeds, add a new physical node to the topology network and update the information of the node set in the dynamic topology node layer.

[0013] Preferably, randomly eliminate the sample data of several training nodes, determine the quality of the sample data of the several training nodes based on the degree of change in the difference between the model output and the true value, and based on the determined result, uniformly reduce or increase the data weights of the participants involved in the training nodes.

[0014] In a second aspect, the embodiments of the present disclosure further provide a database service platform based on an AI large model, including:

[0015] A platform construction function module, which is used to divide the central computing layer, the dynamic topology node layer and the user layer; receive the user node registration information of the user layer and return a user token and the information of the node set in the dynamic topology node layer to the user node;

[0016] The data upload function module is used for non-autonomous user nodes at the user layer to generate a data upload mapping table based on the node set information and a random algorithm. The data upload mapping table contains user token random digest information, and the data upload mapping table is used to map nodes in the node set of the dynamic topology node layer where the user layer data is stored; the user layer uploads user data slices to the mapped nodes based on the data upload mapping table.

[0017] The data training function module. Nodes in the dynamic topology node layer receive user data slices, perform node normalization processing on the user data slices in the data format corresponding to the nodes, perform obfuscation processing on the data slices after node normalization processing, and store them in the database sub-node; the training sub-node trains the sample data in the database sub-node based on the configured AI model and sends the trained model parameters to the central computing layer; autonomous nodes at the user layer train their sample data based on the configured AI model and send the trained model parameters to the central computing layer.

[0018] The parameter fusion function module. The central computing layer receives the model parameters of each node in the dynamic topology node layer and the model parameters of autonomous nodes at the user layer, performs weighted average calculation on the model parameters, and obtains the aggregated total model parameters.

[0019] Preferably, the user slices the user data to be uploaded by unit. The slicing algorithm is a random generation algorithm, which ensures that the minimum slicing unit is the data unit that can be processed by training; the non-autonomous node generates a data upload mapping table based on the received node set information of the dynamic topology node layer and the random algorithm. The data upload mapping table is used to map nodes in the node set of the dynamic topology node layer where the user layer data is stored, and adapts the user data slices and the corresponding upload paths, so as to upload the data slices according to the upload paths of the nodes in the mapped node set of the dynamic topology node layer.

[0020] Preferably, the implementation method of the obfuscation processing is: (1) temporarily store the received data slices in a non-decrypted state; (2) within a preset time, judge whether the data source of the data slices is less than a predetermined quantity threshold. If so, decrypt the data slices and add controllable noise data, and store them in the database sub-node; if not, perform obfuscation processing on the data in the data slices from different data sources and store them in the database sub-node.

[0021] Preferably, the dynamic topology node layer includes a specific number of distributed physical nodes. Within a preset time, it is determined whether the number of nodes with the data source of data slices less than a predetermined quantity threshold exceeds a second predetermined quantity threshold. If it exceeds, several of these nodes are selected for shutdown processing. Before shutdown, their data is dynamically migrated to other nodes to release the physical node. At the same time, within the preset time, it is determined whether the number of nodes with the data source of data slices greater than a third predetermined quantity threshold exceeds a fourth predetermined quantity threshold. If it exceeds, new physical nodes are added to the topology network to update the node set information of the dynamic topology node layer.

[0022] Preferably, sample data of several training nodes is randomly excluded, and the quality of the sample data of the several training nodes is determined based on the degree of change in the difference between the model output and the true value. Based on the determined results, the data weights of the parties involved in the training nodes are uniformly reduced or increased.

[0023] The present invention provides a database service platform and a usage method based on an AI large model, achieving at least the following technical effects: (1) User nodes are divided into autonomous nodes and non-autonomous nodes, and the data processing of non-autonomous nodes is agented by the nodes in the dynamic topology node layer. This creative distributed training meets the comprehensive requirements of each training participant for the training environment, efficiency, and cost; (2) Data is jointly privacy-processed from both the data uploader and the data receiver aspects. At the same time, during the privacy processing, sample data with a small sample size is fused. Compared with the training of samples with a small sample size in the prior art, it not only maximally ensures the privacy and security of the data but also improves the model training efficiency; (3) An innovative training architecture for data processing agency based on the nodes in the dynamic topology node layer is proposed, centralizing the data of non-autonomous node participants, which not only improves the training efficiency but also facilitates ensuring the load balance of training nodes, and solves a series of problems of high cost and low efficiency of small and micro participants in the prior art; (4) A data sample weight adjustment method is set to ensure the output accuracy of the model and the user privacy security. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] By describing the exemplary embodiments of the present invention in more detail in conjunction with the accompanying drawings, the above and other objects, features, and advantages of the present invention will become more obvious. Among them, in the exemplary embodiments of the present invention, the same reference numerals generally represent the same components.

[0025] Figure 1 FIG. shows a flowchart of the steps of a usage method of a database service platform based on an AI large model according to an embodiment of the present invention.

[0026] Figure 2 FIG. shows a block diagram of a database service platform based on an AI large model according to an embodiment of the present invention. Detailed implementation manners

[0027] To make the objectives, technical solutions, and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Apparently, the described embodiments are some, but not all, of the embodiments of the present application. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present application without creative efforts shall fall within the protection scope of the present application.

[0028] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present application and the above-mentioned accompanying drawings are used to distinguish similar objects and do not necessarily need to be used to describe a specific order or sequence. It should be understood that such data can be interchanged under appropriate circumstances so that the embodiments of the present application described herein can be implemented in an order different from those illustrated or described herein. In addition, the terms "include" and "correspond to" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products, or devices.

[0029] To make the objectives, technical solutions, and advantages of the present invention clearer and more understandable, the present invention will be further described in detail below with reference to embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0030] Embodiment 1: As Figure 1 shown, the present invention provides a method for using a database service based on an AI large model, and the method includes the steps of:

[0031] S100, dividing the database service platform into a central computing layer, a dynamic topology node layer, and a user layer; the database service platform receives user node registration information from the user layer and returns a user token and node set information of the dynamic topology node layer to the user node;

[0032] The user layer consists of at least one user node; the user node includes an autonomous node and a non-autonomous node. The dynamic topology node serves as the intermediate layer between the central computing layer and the user layer in the form of a set. On the one hand, it receives the data uploaded by the non-autonomous nodes in the user layer and performs training-related processing. On the other hand, it sends the results obtained from the training-related processing to the central computing layer for further processing. Among them, the autonomous node refers to the node where the model participant manages the user data and model training by itself, and the non-autonomous node refers to the node that uploads the user data to the dynamic topology node layer and uses it as an agent node for training. Among the users who hope to participate in model training, not all users have the software and hardware environment for user data management and model training. At the same time, for some small and micro enterprises, small data volume units or private type users, the cost of fully configuring the training environment is high, and the model training results of small data volumes are also unreliable. Therefore, considering effectiveness, security, and cost, the user nodes are divided into autonomous nodes and non-autonomous nodes. As an important asset, data resources, any part of the data contributes significantly to model algorithm training. The total data possessed by autonomous nodes and non-autonomous nodes constitutes the big data basis of the model.

[0033] The user requests registration from the database service platform to join the AI large model training. The database service platform receives the user node registration information of the user layer. After registration verification, it returns the user token and the node set information of the dynamic topology node layer to the user, and maintains the user information database in the system to record the full user information. Among them, the returned user token is a sequence identifier used to identify the unique identity information of the user, and the node set of the dynamic topology node layer is the dynamic topology nodes that the user has the right to use. The user node realizes related functional operations such as data upload, privacy processing, and data obfuscation based on the node set information. As will be known in the following content, the topology architecture of the dynamic topology node layer changes dynamically, and this change information can be synchronized to the user layer nodes based on various well-known methods. The data in the user information database is used for user management, including basic information management, membership management, permission management, contribution management, etc.

[0034] S110, the non-autonomous user node of the user layer generates a data upload mapping table based on the node set information and a random algorithm. The data upload mapping table contains the random digest information of the user token. The data upload mapping table is used to map the nodes in the node set of the dynamic topology node layer where the user layer data is stored; the user layer uploads the user data slices to the mapped nodes based on the data upload mapping table.

[0035] Privacy is a basic condition for large AI models during the joint training process. The data uploaded by users should ensure its privacy. During the process of uploading data from non-autonomous nodes to the dynamic topology node layer, since the dynamic topology node layer is not managed locally by users, there will inevitably be a risk of data exposure. Therefore, in this step, the data transmission process is dominated by the user side. Specifically, in a preferred embodiment, the user slices the user data to be uploaded into units. The slicing algorithm can be a random generation algorithm, and it should ensure that the minimum slicing unit is a data unit that can be processed by training. Correspondingly, the non-autonomous node generates a data upload mapping table based on the dynamic topology node layer node set information received in step S100 and the random algorithm. The data upload mapping table contains user token random digest information, which is used to verify the permissions and legality of the user's data upload behavior. The data upload mapping table is used to map the nodes in the dynamic topology node layer node set where the user layer data is stored, adapt the user data slices and their corresponding upload paths, and thus upload the data slices according to the upload paths of the nodes in the mapped dynamic topology node layer node set. In the above method, randomly slicing the data and determining the nodes in the dynamic topology node layer as proxy nodes based on a random algorithm hides the integrity of the data and the data processing party, ensuring the privacy of the training sample data.

[0036] In a preferred embodiment, the generation algorithm of the data upload mapping table further includes: (1) User-defined configuration, in which fuzzy proxy node conditions are determined, including the geographical location of storage nodes, load conditions, security levels, permission levels, etc.; (2) Randomly generating mapping nodes with the user-defined configuration as a constraint.

[0037] S120. The nodes in the dynamic topology node layer receive the user data slices, perform node standardization processing on the user data slices in the data format corresponding to this node, perform fusion and confusion processing on the data slices after node standardization processing, and store them in the database sub-node; the training sub-node trains the sample data in the database sub-node based on the configured AI model and sends the trained model parameters to the central computing layer; the autonomous nodes in the user layer train their sample data based on the configured AI model and send the trained model parameters to the central computing layer.

[0038] Each node in the dynamic topology node layer includes a database sub-node and a training sub-node. The database sub-node stores the sample big data for training, and the training sub-node deploys an AI model to perform model training on the sample big data. The dynamic topology node layer contains a specific number of distributed physical nodes. When the average node load exceeds a predetermined value, physical nodes can be newly added to the topology network. When the model training result of a specific node does not meet the convergence condition, its data is dynamically migrated to other nodes to release this physical node.

[0039] The training content of different nodes is different, and this difference is determined by the database service platform according to the training algorithm of the deployed model at the initial stage of environment construction. According to the difference in training content, after receiving the user data slice, the nodes in the dynamic topology node layer perform node standardization processing on the user data slice in the data format corresponding to this node.

[0040] In a preferred embodiment, before storing the received user data slice into the database sub-node, the nodes in the dynamic topology node layer perform fusion and obfuscation processing on the data slice after node standardization processing. The principle of obfuscation processing is to obscure the source of user data. Specifically, its implementation method is: (1) temporarily store the received data slice in a non-decrypted state; (2) within a preset time, determine whether the data source of the data slice is less than a predetermined quantity threshold. If so, decrypt the data slice and add controllable noise data, and then store it in the database sub-node; if not, perform obfuscation processing on the data in the data slices from different data sources and store it in the database sub-node. Before training with application samples, each node in the dynamic topology node layer performs obfuscation processing on the data slice uploaded by the user, eliminating the possibility of tracing the data source, and ensuring data privacy and security from the perspective of the recipient.

[0041] In a preferred embodiment, the obfuscation processing includes: randomly scrambling in units of data slices, and extracting and fusing the data content in the data slices. The non-decrypted state means temporarily relieving the decryption processing after receiving encrypted data, or performing encryption processing after receiving data, so as to ensure the security during data temporary storage. The controllable noise data includes: data containing specific fields for noise filtering, adding fields that are meaningless for model training, changing field mapping, and encrypting fields.

[0042] Furthermore, in a preferred embodiment, the dynamic topology node layer includes a specific number of distributed physical nodes. Within a preset time, determine whether the number of nodes whose data source of the data slice is less than the predetermined quantity threshold exceeds the second predetermined quantity threshold. If it exceeds, select several of them for shutdown processing, and dynamically migrate their data to other nodes before shutdown to release this physical node; at the same time, within a preset time, determine whether the number of nodes whose data source of the data slice is greater than the third predetermined quantity threshold exceeds the fourth predetermined quantity threshold. If it exceeds, add new physical nodes to the topology network and update the node set information of the dynamic topology node layer. Even if it is controllable noise, the data after noise addition processing will still be distorted to a certain extent and introduce unnecessary data processing load; the number of data slice sources is an indication of node load; moreover, if the data volume in the dynamic topology node layer is too small, the training result is not easy to converge or is inaccurate, which will in turn hinder the training process. Therefore, for the above reasons, the dynamic topology node layer is dynamically adjusted, improving the training efficiency of the system and ensuring data privacy and security.

[0043] The training sub-nodes train the sample data in the database sub-nodes based on the configured AI model, and send the trained model parameters to the central computing layer. These training results reflect the training data contribution of the non-autonomous nodes; the autonomous nodes in the user layer train their sample data based on the AI model they configure, and send the trained model parameters to the central computing layer, reflecting the training data contribution of the autonomous nodes.

[0044] S130, the central computing layer receives the model parameters of each node in the dynamic topology node layer and the model parameters of the autonomous nodes in the user layer, and performs weighted average calculation on the model parameters to obtain the aggregated total model parameters.

[0045] The non-autonomous nodes achieve model training under the premise of data privacy and security with the nodes in the dynamic topology node layer as proxy nodes. However, due to the uncontrollability of the proxy process of data transmission and processing, this privacy and security is relative. Therefore, the data processing method of the autonomous nodes is still an option that cannot be discarded. The database service platform of the present invention provides database services to users. It differentiates user nodes into autonomous nodes and non-autonomous nodes, constructs a distributed machine learning framework formed by the nodes in the dynamic topology node layer and the autonomous nodes, so that the autonomous nodes and non-autonomous nodes form a distributed machine learning framework, allowing multiple participants (such as mobile devices, enterprises or data centers) to collaboratively train a global model without sharing the original data. In a preferred embodiment, when obtaining the aggregated model parameters, the ratio of the data load of a node to the total data volume of each node in the dynamic topology node layer and the autonomous nodes in the user layer is used as the weight to perform weighted average calculation on the model parameters.

[0046] In a preferred embodiment, the accuracy of the real-time evaluation model is evaluated. Before parameter fusion, sample data of several nodes are randomly excluded, and the quality of the sample data is determined based on the change in the difference between the model output and the true value. Based on the determined results, the data weights of the involved parties are reduced or increased. By controlling the node data as described above, the probability of model distortion is reduced. In a preferred embodiment, the reduction or increase of the data weights of the involved parties specifically means: determining the data source of the sample data of the training nodes, and setting training weights for the data of the data source nodes. Since the data of the training nodes comes from non-autonomous user nodes, the quality of the sample data of the training nodes is essentially determined by the data slices of the non-autonomous user nodes. Therefore, the optimal way to process the weights of the training samples is to directly process the data source. In a preferred embodiment, to further ensure user privacy, setting the training weights for the data of the data source nodes means: performing the same unified processing of reducing or increasing the weights on all the data sources of the sample data involved in the determined training nodes. From the perspective of a single training node, the unified processing may seem unfair to the nodes that do not need to adjust the sample weights. However, from the overall perspective, since each training node involves user nodes from multiple data sources, the weight adjustment for all training nodes will ultimately accurately affect the user nodes of each data source. For example, the data of training node A comes from non-autonomous nodes a, b, and c, the data of training node B comes from non-autonomous nodes b, c, and d, and the data of training node C comes from non-autonomous nodes a, c, and d. The original weights are a = b = c = d = 100%. Periodically adjust the weights of A, that is, adjust the weights of the non-autonomous nodes a, b, and c involved in it to -10%, then the weight values are a = 90%, b = 90%, c = 90%, and d = 100%; periodically adjust the weights of B, that is, adjust the weights of the non-autonomous nodes b, c, and d involved in it to +5%, then the weight values are a = 90%, b = 95%, c = 95%, and d = 105%; periodically adjust the weights of C, that is, adjust the weights of the non-autonomous nodes a, c, and d involved in it to -10%, then the weight values are a = 80%, b = 95%, c = 85%, and d = 95%. It can be seen that through the overall weight adjustment of all training nodes, the quality of the data of each non-autonomous user node of the actual data source is reflected. This weight adjustment method first ensures the superiority and inferiority of the training sample data, so that the model training result will not be distorted and improves the model accuracy; secondly, the unified processing is performed on all the data sources of the sample data involved in the same training node, which is simple and efficient. Since this method does not distinguish the data sources, it greatly ensures the privacy security of users.

[0047] Embodiment 2: As Figure 2As shown in the figure, the present invention also provides a database service platform based on an AI large model, including:

[0048] A platform construction functional module, which is used to divide the central computing layer, the dynamic topology node layer and the user layer; receive the user node registration information of the user layer, and return the user token and the node set information of the dynamic topology node layer to the user node;

[0049] A data upload functional module, which is used for non-autonomous user nodes in the user layer to generate a data upload mapping table based on the node set information and a random algorithm, the data upload mapping table contains user token random digest information, and the data upload mapping table is used to map the nodes in the node set of the dynamic topology node layer where the user layer data is stored; the user layer uploads user data slices to the mapped nodes based on the data upload mapping table;

[0050] A data training functional module, the nodes in the dynamic topology node layer receive the user data slices, perform node standardization processing on the user data slices in the data format corresponding to the node, perform confusion processing on the data slices after the node standardization processing, and store them in the database sub-node; the training sub-node trains the sample data in the database sub-node based on the configured AI model, and sends the trained model parameters to the central computing layer; the autonomous nodes in the user layer train their sample data based on the configured AI model, and send the trained model parameters to the central computing layer;

[0051] A parameter fusion functional module, the central computing layer receives the model parameters of each node in the dynamic topology node layer and the model parameters of the autonomous nodes in the user layer, performs weighted average calculation on the model parameters, and obtains the aggregated total model parameters.

[0052] In a preferred embodiment, the user slices the user data to be uploaded by unit, and the slicing algorithm is a random generation algorithm, which ensures that the minimum slicing unit is the data unit that can be processed by training; the non-autonomous node generates a data upload mapping table based on the received node set information of the dynamic topology node layer and the random algorithm, the data upload mapping table is used to map the nodes in the node set of the dynamic topology node layer where the user layer data is stored, adapts the user data slices and the corresponding upload paths, and thus uploads the data slices according to the upload paths of the nodes in the mapped node set of the dynamic topology node layer.

[0053] In a preferred embodiment, the implementation method of the confusion processing is: (1) temporarily store the received data slices in a non-decrypted state; (2) within a preset time, judge whether the data source of the data slices is less than a predetermined quantity threshold, if so, decrypt the data slices and add controllable noise data, and store them in the database sub-node; if not, perform confusion processing on the data in the data slices from different data sources, and store them in the database sub-node.

[0054] In a preferred embodiment, the dynamic topology node layer includes a specific number of distributed physical nodes. Within a preset time, it is determined whether the number of nodes with the data source of data slices less than a predetermined quantity threshold exceeds a second predetermined quantity threshold. If it exceeds, several of these nodes are selected for shutdown processing. Before shutdown, their data is dynamically migrated to other nodes to release the physical node. At the same time, within the preset time, it is determined whether the number of nodes with the data source of data slices greater than a third predetermined quantity threshold exceeds a fourth predetermined quantity threshold. If it exceeds, new physical nodes are added to the topology network to update the node set information of the dynamic topology node layer.

[0055] In a preferred embodiment, sample data of several training nodes is randomly excluded, and the quality of the sample data of the several training nodes is determined based on the degree of change in the difference between the model output and the true value. Based on the determined result, the data weights of the parties involved in the training nodes are uniformly reduced or increased.

[0056] The present invention provides a database service platform based on an AI large model and a usage method, achieving at least the following technical effects: (1) User nodes are divided into autonomous nodes and non-autonomous nodes. The data processing of non-autonomous nodes is proxied by nodes in the dynamic topology node layer. This creative distributed training meets the comprehensive requirements of each training participant for the training environment, efficiency, and cost; (2) Data privacy processing is jointly performed from two aspects: the data uploader and the data receiver. At the same time, during the privacy processing, sample data with a small sample size is fused. Compared with the training of samples with a small sample size in the prior art, it not only maximally ensures the privacy and security of data but also improves the model training efficiency; (3) An innovative training architecture for data processing proxy based on nodes in the dynamic topology node layer is proposed, which centralizes the data of non-autonomous node participants, improving the training efficiency, facilitating the load balancing of training nodes, and solving a series of problems of high cost and low efficiency of small and micro participants in the prior art; (4) A data sample weight adjustment method is set to ensure the output accuracy of the model and the privacy and security of users.

[0057] According to one embodiment, a program product such as a machine-readable medium is provided. The machine-readable medium may have instructions (i.e., the elements implemented in software as described above), which when executed by the machine, cause the machine to perform the various operations and functions described above in the respective embodiments of this specification. Specifically, a system or device equipped with a readable storage medium may be provided, on which software program code for implementing the functions of any one of the above embodiments is stored, and the computer or processor of the system or device reads and executes the instructions stored in the readable storage medium. Figure 1

[0058] ​In this case, the program code read from the readable medium itself can implement the functions of any one of the above embodiments. Therefore, the machine-readable code and the readable storage medium storing the machine-readable code constitute a part of this specification.

[0059] Examples of the readable storage medium include floppy disks, hard disks, magneto-optical disks, optical disks (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD-RW), magnetic tapes, non-volatile memory cards, and ROMs. Optionally, the program code can be downloaded from a server computer or a cloud via a communication network.

[0060] Those skilled in the art should understand that various modifications and variations can be made to the above-disclosed embodiments without departing from the essence of the invention. Therefore, the protection scope of this specification should be defined by the appended claims.

[0061] It should be noted that not all steps and units in the above-mentioned processes and system structure diagrams are necessary, and some steps or units can be omitted according to actual needs. The execution order of the steps is not fixed and can be determined as needed. The device structures described in the above embodiments can be physical structures or logical structures. That is, some units may be implemented by the same physical entity, or some units may be implemented separately by multiple physical entities, or some components in multiple independent devices may be jointly implemented.

[0062] In the above embodiments, the hardware units or modules can be implemented mechanically or electrically. For example, a hardware unit, module, or processor can include permanent dedicated circuits or logic (such as a dedicated processor, FPGA, or ASIC) to perform corresponding operations. The hardware unit or processor can also include programmable logic or circuits (such as a general-purpose processor or other programmable processors), which can be temporarily set by software to perform corresponding operations. The specific implementation method (mechanical method, or dedicated permanent circuit, or temporarily set circuit) can be determined based on cost and time considerations.

[0063] The specific embodiments described above in conjunction with the accompanying drawings describe exemplary embodiments, but do not represent all embodiments that can be implemented or fall within the protection scope of the claims. The term "exemplary" used throughout this specification means "serving as an example, instance, or illustration", and does not mean "preferred" or "superior" to other embodiments. For the purpose of providing an understanding of the described technology, the specific embodiments include specific details. However, these technologies can be implemented without these specific details. In some instances, well-known structures and devices are shown in block diagram form to avoid obscuring the concepts of the described embodiments.

[0064] The foregoing description of the present disclosure is provided to enable any person of ordinary skill in the art to make or use the present disclosure. Various modifications to the present disclosure will be apparent to those of ordinary skill in the art, and the general principles corresponding thereto herein can also be applied to other variations without departing from the scope of protection of the present disclosure. Therefore, the present disclosure is not limited to the examples and designs described herein, but is consistent with the broadest scope that conforms to the principles and novel features disclosed herein.

Claims

1. A method for using a database service platform based on an AI large model, characterized in that The method includes the steps of: S100, dividing the database service platform into a central computing layer, a dynamic topology node layer, and a user layer; the database service platform receives user node registration information from the user layer and returns a user token and node set information of the dynamic topology node layer to the user node; S110, the non-autonomous user nodes in the user layer generate a data upload mapping table based on the node set information and a random algorithm. The data upload mapping table includes user token random digest information, and the data upload mapping table is used to map the nodes in the node set of the dynamic topology node layer where the user layer data is stored; the user layer uploads user data slices to the mapped nodes based on the data upload mapping table; S120, the nodes in the dynamic topology node layer receive the user data slices, perform node standardization processing on the user data slices in the data format corresponding to this node, perform obfuscation processing on the data slices after node standardization processing, and store them in the database sub-nodes; the training sub-nodes train the sample data in the database sub-nodes based on the configured AI model and send the trained model parameters to the central computing layer; the autonomous nodes in the user layer train their sample data based on the AI model they configure and send the trained model parameters to the central computing layer; S130, the central computing layer receives the model parameters of each node in the dynamic topology node layer and the model parameters of the autonomous nodes in the user layer, and performs weighted average calculation on the model parameters to obtain the aggregated total model parameters.

2. The method according to claim 1, characterized in that: The user slices the user data to be uploaded by a slicing algorithm by unit. The slicing algorithm is a random generation algorithm to ensure that the minimum slicing unit is a data unit that can be processed by training; The non-autonomous nodes generate a data upload mapping table based on the received node set information of the dynamic topology node layer and a random algorithm. The data upload mapping table is used to map the nodes in the node set of the dynamic topology node layer where the user layer data is stored, and adapt the user data slices and the corresponding upload paths, so as to upload the data slices according to the upload paths of the nodes in the mapped node set of the dynamic topology node layer.

3. The method according to claim 1, wherein: The obfuscation processing method includes: Temporarily storing the received data slices in a non-decrypted state; Within a preset time, judge whether the data source of the data slice is less than a predetermined quantity threshold. If so, decrypt the data slice and add controllable noise data, and store it in the database sub-node; if not, perform obfuscation processing on the data in the data slices from different data sources and store it in the database sub-node.

4. The method according to claim 1, wherein: The dynamic topology node layer includes a specific number of distributed physical nodes. Within a preset time, judge whether the number of nodes whose data source of the data slice is less than the predetermined quantity threshold exceeds a second predetermined quantity threshold. If it exceeds, select several of them for shutdown processing, and dynamically migrate their data to other nodes before shutdown to release the physical node; And within a preset time, judge whether the number of nodes whose data source of the data slice is greater than a third predetermined quantity threshold exceeds a fourth predetermined quantity threshold. If it exceeds, add new physical nodes to the topology network and update the node set information of the dynamic topology node layer.

5. The method according to claim 1, wherein: Randomly eliminate the sample data of several training nodes, determine the quality of the sample data of the several training nodes based on the degree of change in the difference between the model output and the true value, and uniformly reduce or increase the data weights of the parties involved in the training nodes based on the determined results.

6. A database service platform based on a large AI model, characterized in that, The platform includes: A platform construction function module for dividing the central computing layer, the dynamic topology node layer, and the user layer; receiving user node registration information from the user layer and returning user tokens and dynamic topology node layer node set information to the user nodes. A data upload function module for non-autonomous user nodes in the user layer to generate a data upload mapping table based on the node set information and a random algorithm. The data upload mapping table includes random digest information of the user token, and the data upload mapping table is used to map the nodes in the dynamic topology node layer node set stored by the user layer data; the user layer uploads user data slices to the mapped nodes based on the data upload mapping table. A data training function module. The nodes in the dynamic topology node layer receive the user data slices, perform node standardization processing on the user data slices in the data format corresponding to the nodes, perform confusion processing on the data slices after node standardization processing, and store them in the database sub-nodes; the training sub-nodes train the sample data in the database sub-nodes based on the configured AI model and send the trained model parameters to the central computing layer; the autonomous nodes in the user layer train their sample data based on the configured AI model and send the trained model parameters to the central computing layer. A parameter fusion function module. The central computing layer receives the model parameters of each node in the dynamic topology node layer and the model parameters of the autonomous nodes in the user layer, performs weighted average calculation on the model parameters, and obtains the aggregated total model parameters.

7. The database service platform based on the AI large model according to claim 6, characterized in that: The user slices the user data to be uploaded using a slicing algorithm. The slicing algorithm is a random generation algorithm to ensure that the minimum slicing unit is a data unit that can be processed by training. The non-autonomous nodes generate a data upload mapping table based on the received dynamic topology node layer node set information and a random algorithm. The data upload mapping table is used to map the nodes in the dynamic topology node layer node set stored by the user layer data, and adapt the user data slices and the corresponding upload paths, so as to upload the data slices according to the upload paths of the nodes in the mapped dynamic topology node layer node set.

8. The database service platform based on the AI large model according to claim 6, wherein: The confusion processing method includes: temporarily storing the received data slices in a non-decrypted state; within a preset time, determining whether the data source of the data slices is less than a predetermined quantity threshold. If so, decrypt the data slices and add controllable noise data, and store them in the database sub-nodes; if not, perform confusion processing on the data in the data slices from different data sources and store them in the database sub-nodes.

9. The database service platform based on the AI large model as described in claim 6, wherein the dynamic topology node layer includes a specific number of distributed physical nodes. Within a preset time, it is determined whether the number of nodes with the data source of the data slice less than a predetermined quantity threshold exceeds a second predetermined quantity threshold. If it exceeds, several of these nodes are selected for shutdown processing. Before shutdown, their data is dynamically migrated to other nodes, and the physical node is released. And within a preset time, it is determined whether the number of nodes with the data source of the data slice greater than a third predetermined quantity threshold exceeds a fourth predetermined quantity threshold. If it exceeds, new physical nodes are added to the topology network, and the node set information of the dynamic topology node layer is updated.

10. The database service platform based on the AI large model according to claim 6, characterized in that: Randomly eliminate the sample data of several training nodes, determine the quality of the sample data of the several training nodes based on the degree of change in the difference between the model output and the true value. Based on the determined result, uniformly reduce or increase the data weights of the parties involved in the training nodes.

Images