Data processing method and device

Abstract

The invention discloses a data processing method and device, and the method comprises the steps: storing a first type of feature vector table in a plurality of feature vector tables to each equipment node in a plurality of equipment nodes, storing a plurality of feature vectors in at least one feature vector table in a second type of feature vector tables in the plurality of feature vector tables to different device nodes in the plurality of device nodes, wherein the feature vectors stored in the different device nodes are different; wherein the sum of the data storage capacity corresponding to the first type of feature vector table and the data storage capacity corresponding to the second type of feature vector table is smaller than or equal to the total data storage capacity of the multiple device nodes. The data storage capacity corresponding to the first-class feature vector table is the product of the total data volume of the first-class feature vector table and the number of the multiple equipment nodes, and the data storage capacity corresponding to the second-class feature vector table is the total data volume of the second-class feature vector table. According to the method, the processing efficiency of the feature vector table and the balance of equipment node loads can be improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and in particular to a data processing method and device. Background Art

[0002] In the fields of artificial intelligence (AI), machine learning (ML), etc., the recommendation function can be realized by training the recommendation model. The training system for training the recommendation model usually consists of a host node and multiple device nodes. The host node can be used for training deployment and splitting the feature vector table (embedding table) according to certain rules and storing it in the device node. The device node can be used to store the feature vector table data assigned by the host node, and to perform model training according to the training data. Among them, the feature vector table is a collection of feature vectors (embedding), the feature vector is a one-dimensional or multi-dimensional data vector, and a feature vector can be used to represent a sample data. The training data used in the model training process is a collection of feature indexes (feature ID), wherein a feature index can be used to index a row of feature vectors in the feature vector table. The process of the device node performing model training according to the training data includes: first, determining the feature vector required for model training according to the feature index included in the training data, then obtaining the required feature vector from the device node and / or other device nodes, and finally performing model training according to the obtained feature vector.

[0003] The currently widely used master node splits the feature vector table and stores it in device nodes in the following ways: column-wise, row-wise, and data-parallel. The row-wise method is to divide the feature vector table into rows and store the divided data in different device nodes. The column-wise method is to divide the feature vector table into columns and store the divided data in different device nodes. The data-parallel method is to copy the feature vector table and store it in multiple device nodes, that is, each device node stores a copy of the feature vector table.

[0004] In practical applications, the method used to process the feature vector table can be manually selected or automatically selected from the above multiple methods. Among them, the efficiency of manual selection is low, and it is difficult to obtain a high training efficiency due to the experience of the processing personnel. In the automatic selection method, the master node can automatically select the feature vector table processing method based on the business scale and device hardware parameters, but this method may cause unbalanced communication load of device nodes, resulting in waste of computing power of some device nodes. Therefore, the training efficiency of the automatic selection method is also low. Summary of the invention

[0005] The present application provides a data processing method and device to improve the efficiency of processing feature vector tables, improve the balance of device node loads, and thereby improve the efficiency of model training.

[0006] In a first aspect, the present application provides a data processing method, the method comprising: obtaining multiple feature vector tables to be stored; wherein each feature vector table in the multiple feature vector tables includes one or more feature vectors; determining a first type of feature vector table and a second type of feature vector table in the multiple feature vector tables; wherein the sum of the data storage capacity corresponding to the first type of feature vector table and the data storage capacity corresponding to the second type of feature vector table is less than or equal to the total data storage capacity of multiple device nodes, the data storage capacity corresponding to the first type of feature vector table is the product of the total data volume of the first type of feature vector table and the number of the multiple device nodes, and the data storage capacity corresponding to the second type of feature vector table is the total data volume of the second type of feature vector table; storing the first type of feature vector table in each device node of the multiple device nodes respectively; and storing multiple feature vectors in at least one feature vector table included in the second type of feature vector table in different device nodes of the multiple device nodes, and the feature vectors stored in different device nodes are different.

[0007] In this method, the storage method of the first type of feature vector table includes a data parallel method, which does not have a communication load between device nodes, can alleviate the problem of unbalanced communication load between device nodes when storing feature vector tables, and can also reduce the communication between device nodes during model training, thereby improving the efficiency of model training, but this method requires a large storage space. The storage method of the second type of feature vector table includes a row segmentation method or a column segmentation method, which requires a small storage space and can save storage space, but there is a certain communication load between device nodes. Therefore, in the above method, the master node can divide the multiple feature vector tables to be stored into the first type of feature vector table and the second type of feature vector table according to the data volume of the multiple feature vector tables to be stored and the data storage capacity of the multiple device nodes, and then use the corresponding storage method to store and process the first type of feature vector table and the second type of feature vector table, respectively, which can more fully and effectively utilize the storage space of multiple device nodes on the one hand, and can appropriately reduce the degree of unbalanced communication load between device nodes on the other hand, thereby reducing the waste of computing power of device nodes and improving the efficiency of model training. In addition, the master node can adaptively store the feature vector table according to the above method, and the flexibility, practicality and processing efficiency are all high.

[0008] In a possible design, the data volume of the feature vector table with the smallest data volume in the second type of feature vector table is greater than or equal to the data volume of the feature vector table with the largest data volume in the first type of feature vector table. The first type of feature vector table determined by the master node based on the method includes at least one feature vector table with a smaller data volume in multiple vector tables, and the second type of feature vector table determined based on the method includes at least one feature vector table with a larger data volume in multiple vector tables, which can ensure that the storage method corresponding to the first type of feature vector table is used to process the feature vector table with a smaller data volume as much as possible, thereby increasing the number of feature vector tables that can be processed using the storage method corresponding to the first type of feature vector table, thereby maximizing data access efficiency.

[0009] In one possible design, the product of the first data amount and the number of the multiple device nodes is greater than the difference between the total data storage capacity of the multiple device nodes and the second data amount; wherein the first data amount is the data amount of the first feature vector table, the first feature vector table is the feature vector table with the smallest data amount in the second type of feature vector table, and the second data amount is the total data amount of other feature vector tables in the second type of feature vector table except the first feature vector table.

[0010] Through this method, the master node can maximize the number of feature vector tables processed using the storage method corresponding to the first type of feature vector table, while the total data storage capacity required to store multiple feature vector tables does not exceed the data storage capacity of the device node, which can minimize the communication load between device nodes, maximize data access efficiency, and thus maximize model training efficiency. At the same time, this method can make full use of the storage space of multiple device nodes.

[0011] In one possible design, determining the first and second types of feature vector tables in the multiple feature vector tables includes: repeatedly executing the division process until a termination condition is met to obtain the second type of feature vector table; the division process includes: using the feature vector table in the multiple feature vector tables that is not divided into the second type of feature vector table as a reference table; determining first reference data, wherein the first reference data is the difference between the total data storage capacity of the multiple device nodes and a third data volume, and the third data volume is the total data volume of the feature vector tables that have been divided into the second type of feature vector tables; when the total data volume of the reference table is greater than the first reference data, dividing the feature vector table with the largest data volume in the reference table into the second type of feature vector table; the termination condition is that the total data volume of the reference table is less than or equal to the first reference data; dividing the feature vector table in the multiple feature vector tables that is not divided into the second type of feature vector table into the first type of feature vector table.

[0012] Through this method, the main node can select at least one feature vector table with the largest data volume from multiple feature vector tables, and divide the at least one feature vector table into a second-category feature vector table. The main node can select at least one feature vector table with the smallest data volume from multiple feature vector tables, and divide the at least one feature vector table into a first-category feature vector table. Therefore, it is possible to maximize the number of feature vector tables processed using the storage method corresponding to the first-category feature vector table.

[0013] In a possible design, storing multiple feature vectors in at least one feature vector table included in the second type of feature vector table to different device nodes among the multiple device nodes includes: dividing the multiple feature vectors in at least one feature vector table included in the second type of feature vector table into different partial feature vectors; storing the different partial feature vectors to different device nodes among the multiple device nodes. Based on this method, the master node can store the second type of feature vector table in a dispersed manner to multiple device nodes, which occupies less storage space of the device nodes and can save storage space of the device nodes.

[0014] In a possible design, when the first scoring data is less than the second scoring data, the partial feature vector is at least one row of feature vectors; when the first scoring data is greater than the second scoring data, the partial feature vector is at least one column of feature vectors; when the first scoring data is equal to the second scoring data, the partial feature vector is at least one row of feature vectors or at least one column of feature vectors; wherein, the first scoring data is used to indicate: when the partial feature vector is at least one row of feature vectors, how long it takes for the first device node to obtain the target feature vector; the second scoring data is used to indicate: when the partial feature vector is at least one column of feature vectors, how long it takes for the first device node to obtain the target feature vector; the target feature vector includes one or more rows of feature vectors in a second feature vector table, the second feature vector table is any feature vector table in the second type of feature vector table, and the first device node is any device node among the multiple device nodes.

[0015] In this method, the size of the scoring data is used to indicate the length of time required for a device node to obtain a feature vector in a feature vector table. The size of the scoring data is positively correlated with the length of time required, that is, the larger the scoring data, the longer the time required, and the smaller the scoring data, the shorter the time required. The shorter the time required for a device node to obtain a feature vector in a feature vector table, the higher the efficiency. Therefore, the master node can use the processing method with smaller scoring data as the processing method used when processing the feature vector table, thereby improving processing efficiency.

[0016] In one possible design, the method also includes: obtaining first time data; wherein the first time data is used to indicate the length of time required for the first device node to obtain the first data, and the first data includes: when the partial feature vector is at least one row of feature vectors, the feature vector stored in the target feature vector of the first device node; obtaining second time data; wherein the second time data is used to indicate the length of time required for the first device node to obtain the second data, and the second data includes: when the partial feature vector is at least one row of feature vectors, the feature vector stored in the target feature vector of device nodes other than the first device node; determining the first scoring data based on the first time data and the second time data.

[0017] In this method, when the feature vector table is stored in multiple device nodes in a distributed manner by rows, the first time data is used to represent the time required for the device node to obtain local data, and the second time data is used to represent the time required for the device node to obtain data from other device nodes. The master node can more accurately determine the time required for the device node to obtain a certain amount of data based on the first time data and the second time data.

[0018] In one possible design, obtaining the first time data includes: obtaining the data volume of the target data, first probability data, and first rate data; wherein the first probability data is used to indicate the probability that the target feature vector is stored in the first device node, and the first rate data is used to indicate the rate at which the first device node reads local data; determining the data volume of the first data based on the data volume of the target data and the first probability data; determining the first time data based on the data volume of the first data and the first rate data.

[0019] In this method, based on the data volume of the target data and the first probability data, the data volume that the first device node needs to obtain from the local when obtaining the target data, i.e., the data volume of the first data, can be estimated, and based on the data volume of the first data and the first rate data, the time required for the first device node to obtain the first data from the local can be accurately determined. Therefore, based on this method, the time required for the first device node to obtain the data from the local can be estimated, so as to calculate the scoring data based on the obtained data.

[0020] In one possible design, the obtaining of the second time data includes: obtaining the data volume of the target data, second probability data, and second rate data; wherein the second probability data is used to indicate the probability of the target feature vector being stored in a device node other than the first device node, and the second rate data is used to indicate the data transmission rate between the first device node and device nodes other than the first device node; determining the data volume of the second data based on the data volume of the target data and the second probability data; determining the second time data based on the data volume of the second data and the second rate data.

[0021] In this method, based on the data volume of the target data and the second probability data, the data volume that the first device node needs to obtain from other device nodes when obtaining the target data, i.e., the data volume of the second data, can be estimated. Based on the data volume of the second data and the second rate data, the time required for the first device node to obtain the second data from other device nodes can be accurately determined. Therefore, based on this method, the time required for the first device node to obtain data from other device nodes can be estimated, so as to calculate the scoring data based on the obtained data.

[0022] In one possible design, the method also includes: obtaining third time data; wherein the third time data is used to indicate the length of time required for the first device node to obtain the third data, and the third data includes: when the partial feature vector is at least one column of feature vectors, the feature vector stored in the first device node in the target feature vector; obtaining fourth time data; wherein the fourth time data is used to indicate the length of time required for the first device node to obtain the fourth data, and the fourth data includes: when the partial feature vector is at least one column of feature vectors, the feature vector of device nodes other than the first device node stored in the target feature vector; determine the second scoring data based on the third time data and the fourth time data.

[0023] In this method, when the feature vector table is stored in multiple device nodes in a column-dispersed manner, the third time data is used to represent the time required for the device node to obtain local data, and the fourth time data is used to represent the time required for the device node to obtain data from other device nodes. The master node can more accurately determine the time required for the device node to obtain a certain amount of data based on the third time data and the fourth time data.

[0024] In one possible design, obtaining the third time data includes: obtaining the data volume of the target data, the number of the multiple device nodes, first rate data, and a set first coefficient; wherein the first rate data is used to indicate the rate at which the first device node reads local data; determining the data volume of the third data based on the data volume of the target data and the number of the multiple device nodes; determining the third time data based on the data volume of the third data, the first rate data and the first coefficient.

[0025] In this method, the amount of data that the first device node needs to obtain from the local when obtaining the target data, i.e., the amount of the third data, can be calculated based on the amount of the target data and the number of multiple device nodes. The time required for the device node to obtain the third data from the local can be accurately determined based on the amount of the third data and the first rate data. Therefore, based on this method, the time required for the first device node to obtain the data from the local can be estimated, so as to calculate the scoring data based on the obtained data.

[0026] In one possible design, obtaining the fourth time data includes: obtaining the data volume of the target data, the number of the multiple device nodes, second rate data, and the first coefficient; wherein the second rate data is used to indicate the data transmission rate between the first device node and device nodes other than the first device node; determining the data volume of the fourth data based on the data volume of the target data and the number of the multiple device nodes; determining the fourth time data based on the data volume of the fourth data, the second rate data and the first coefficient.

[0027] In this method, based on the data volume of the target data and the number of multiple device nodes, the data volume that the first device node needs to obtain from other device nodes when obtaining the target data, i.e., the data volume of the fourth data, can be calculated, and based on the data volume of the fourth data and the second rate data, the time required for the first device node to obtain the fourth data from other device nodes can be accurately determined. Therefore, based on this method, the time required for the first device node to obtain data from other device nodes can be estimated, so as to calculate the scoring data based on the obtained data.

[0028] In one possible design, the target feature vector is any one of N feature vectors; wherein the N feature vectors constitute the second feature vector table, the second feature vector table is used to train the AI ​​model, the training process of the AI ​​model includes N training stages, the N feature vectors correspond one-to-one to the N training stages, each of the N feature vectors is used as training data for a corresponding training stage, and N is a positive integer. Optionally, before obtaining multiple feature vector tables to be stored, the method further includes: determining the start time of a set period; wherein the set period includes at least one of the training stages.

[0029] Based on this method, the master node can periodically determine the storage mode of the feature vector table during the training process, which helps to adaptively and periodically adjust the storage mode of the feature vector table according to the results of the periodic determination, thereby helping to improve the utilization rate of device storage resources. At the same time, it can basically not affect the model training process of the device node, and the processing efficiency is relatively high.

[0030] In a second aspect, the present application provides a data processing device, which includes: a data acquisition unit, used to acquire multiple feature vector tables to be stored; wherein each feature vector table in the multiple feature vector tables includes one or more feature vectors; a data processing unit, used to determine a first type of feature vector table and a second type of feature vector table in the multiple feature vector tables; wherein the sum of the data storage capacity corresponding to the first type of feature vector table and the data storage capacity corresponding to the second type of feature vector table is less than or equal to the total data storage capacity of multiple device nodes, the data storage capacity corresponding to the first type of feature vector table is the product of the total data volume of the first type of feature vector table and the number of the multiple device nodes, and the data storage capacity corresponding to the second type of feature vector table is the total data volume of the second type of feature vector table; the data processing unit is also used to store the first type of feature vector table in each device node of the multiple device nodes respectively; and, store multiple feature vectors in at least one feature vector table included in the second type of feature vector table in different device nodes among the multiple device nodes, and the feature vectors stored in different device nodes are different.

[0031] In a possible design, the data volume of the feature vector table with the smallest data volume in the second category of feature vector tables is greater than or equal to the data volume of the feature vector table with the largest data volume in the first category of feature vector tables.

[0032] In one possible design, the product of the first data amount and the number of the multiple device nodes is greater than the difference between the total data storage capacity of the multiple device nodes and the second data amount; wherein the first data amount is the data amount of the first feature vector table, the first feature vector table is the feature vector table with the smallest data amount in the second type of feature vector table, and the second data amount is the total data amount of other feature vector tables in the second type of feature vector table except the first feature vector table.

[0033] In a possible design, when the data processing unit determines the first type of feature vector table and the second type of feature vector table in the multiple feature vector tables, it is specifically used to: repeat the division process until the termination condition is met to obtain the second type of feature vector table; the division process includes: taking the feature vector table in the multiple feature vector tables that is not divided into the second type of feature vector table as a reference table; determining first reference data, wherein the first reference data is the difference between the total data storage capacity of the multiple device nodes and the third data volume, and the third data volume is the total data volume of the feature vector tables that have been divided into the second type of feature vector tables; when the total data volume of the reference table is greater than the first reference data, dividing the feature vector table with the largest data volume in the reference table into the second type of feature vector table; the termination condition is that the total data volume of the reference table is less than or equal to the first reference data; dividing the feature vector table in the multiple feature vector tables that is not divided into the second type of feature vector table into the first type of feature vector table.

[0034] In one possible design, when the data processing unit stores multiple feature vectors in at least one feature vector table included in the second type of feature vector table to different device nodes among the multiple device nodes, it is specifically used to: divide the multiple feature vectors in at least one feature vector table included in the second type of feature vector table into different partial feature vectors; and store the different partial feature vectors to different device nodes among the multiple device nodes.

[0035] In a possible design, when the first scoring data is less than the second scoring data, the partial feature vector is at least one row of feature vectors; when the first scoring data is greater than the second scoring data, the partial feature vector is at least one column of feature vectors; when the first scoring data is equal to the second scoring data, the partial feature vector is at least one row of feature vectors or at least one column of feature vectors; wherein, the first scoring data is used to indicate: when the partial feature vector is at least one row of feature vectors, how long it takes for the first device node to obtain the target feature vector; the second scoring data is used to indicate: when the partial feature vector is at least one column of feature vectors, how long it takes for the first device node to obtain the target feature vector; the target feature vector includes one or more rows of feature vectors in a second feature vector table, the second feature vector table is any feature vector table in the second type of feature vector table, and the first device node is any device node among the multiple device nodes.

[0036] In one possible design, the data processing unit is also used to: obtain first time data; wherein the first time data is used to indicate the length of time required for the first device node to obtain the first data, and the first data includes: when the partial feature vector is at least one row of feature vectors, the feature vector stored in the target feature vector of the first device node; obtain second time data; wherein the second time data is used to indicate the length of time required for the first device node to obtain the second data, and the second data includes: when the partial feature vector is at least one row of feature vectors, the feature vectors of device nodes other than the first device node stored in the target feature vector; determine the first scoring data based on the first time data and the second time data.

[0037] In one possible design, when the data processing unit obtains the first time data, it is specifically used to: obtain the data volume of the target data, first probability data, and first rate data; wherein the first probability data is used to indicate the probability that the target feature vector is stored in the first device node, and the first rate data is used to indicate the rate at which the first device node reads local data; determine the data volume of the first data based on the data volume of the target data and the first probability data; determine the first time data based on the data volume of the first data and the first rate data.

[0038] In one possible design, when the data processing unit obtains the second time data, it is specifically used to: obtain the data volume of the target data, second probability data, and second rate data; wherein the second probability data is used to indicate the probability of the target feature vector being stored in a device node other than the first device node, and the second rate data is used to indicate the data transmission rate between the first device node and device nodes other than the first device node; determine the data volume of the second data based on the amount of the target data and the second probability data; determine the second time data based on the data volume of the second data and the second rate data.

[0039] In one possible design, the data processing unit is also used to: obtain third time data; wherein the third time data is used to indicate the length of time required for the first device node to obtain the third data, and the third data includes: when the partial feature vector is at least one column of feature vectors, the feature vector stored in the first device node in the target feature vector; obtain fourth time data; wherein the fourth time data is used to indicate the length of time required for the first device node to obtain the fourth data, and the fourth data includes: when the partial feature vector is at least one column of feature vectors, the feature vector of device nodes other than the first device node stored in the target feature vector; determine the second scoring data based on the third time data and the fourth time data.

[0040] In one possible design, when the data processing unit obtains the third time data, it is specifically used to: obtain the data volume of the target data, the number of the multiple device nodes, the first rate data, and the set first coefficient; wherein the first rate data is used to indicate the rate at which the first device node reads local data; determine the data volume of the third data based on the data volume of the target data and the number of the multiple device nodes; determine the third time data based on the data volume of the third data, the first rate data and the first coefficient.

[0041] In one possible design, when the data processing unit obtains the fourth time data, it is specifically used to: obtain the data volume of the target data, the number of the multiple device nodes, second rate data, and the first coefficient; wherein the second rate data is used to indicate the data transmission rate between the first device node and device nodes other than the first device node; determine the data volume of the fourth data based on the data volume of the target data and the number of the multiple device nodes; determine the fourth time data based on the data volume of the fourth data, the second rate data and the first coefficient.

[0042] In one possible design, the target feature vector is any one of N feature vectors; wherein the N feature vectors constitute the second feature vector table, the second feature vector table is used to train the AI ​​model, the training process of the AI ​​model includes N training stages, the N feature vectors correspond one-to-one to the N training stages, each of the N feature vectors is used as training data for a corresponding training stage, and N is a positive integer.

[0043] In a possible design, before acquiring the multiple feature vector tables to be stored, the data acquisition unit is further used to: determine the start time of a set period; wherein the set period includes at least one of the training stages.

[0044] In one possible design, the data processing device is a master node in an AI training cluster. Optionally, the data processing device is an AI training device.

[0045] In a third aspect, the present application provides a data processing device, comprising a memory and at least one processor; the memory is used to store computer programs; the processor is used to execute the computer programs stored in the memory to implement the method described in the first aspect or any possible design of the first aspect.

[0046] In one possible design, the data processing device is a master node in an AI training cluster. Optionally, the data processing device is an AI training device.

[0047] In a fourth aspect, the present application provides a data processing device, comprising at least one processor and an interface; the interface is used to provide program instructions or data to the at least one processor; the at least one processor is used to execute the program instructions to implement the method described in the above-mentioned first aspect or any possible design of the first aspect.

[0048] In one possible design, the data processing device is a master node in an AI training cluster. Optionally, the data processing device is an AI training device.

[0049] In a fifth aspect, the present application provides a computer-readable storage medium, which stores a computer program. When the computer program runs on a data processing device, the data processing device executes the method described in the first aspect or any possible design of the first aspect.

[0050] In one possible design, the data processing device is a master node in an AI training cluster. Optionally, the data processing device is an AI training device.

[0051] In a sixth aspect, the present application provides a computer program product, comprising a computer program or instructions, which, when executed by a data processing device, implements the method described in the first aspect or any possible design of the first aspect.

[0052] In one possible design, the data processing device is a master node in an AI training cluster. Optionally, the data processing device is an AI training device.

[0053] In the seventh aspect, the present application provides a chip system, which includes at least one processor and an interface, wherein the interface is used to provide program instructions or data to the at least one processor, and the at least one processor is used to execute the program instructions to implement the method described in the above-mentioned first aspect or any possible design of the first aspect.

[0054] In one possible design, the chip system also includes a memory, which is used to store program instructions and data.

[0055] In one possible design, the chip system is composed of a chip, or includes a chip and other discrete devices.

[0056] For the beneficial effects of the second to seventh aspects, please refer to the description of the beneficial effects of the first aspect, which will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 A schematic diagram of a scenario to which a data processing method provided in an embodiment of the present application is applicable;

[0058] Figure 2 A schematic diagram of a system architecture applicable to a data processing method provided in an embodiment of the present application;

[0059] Figure 3 A schematic diagram of a data processing method provided in an embodiment of the present application;

[0060] Figure 4 A schematic diagram of a feature vector table partitioning method provided in an embodiment of the present application;

[0061] Figure 5 A flowchart of a data processing method provided in an embodiment of the present application;

[0062] Figure 6 A schematic diagram of a data processing device provided in an embodiment of the present application;

[0063] Figure 7 A schematic diagram of the structure of a data processing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0064] In order to make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the embodiments of the present application will be further described in detail below in conjunction with the accompanying drawings. Among them, in the description of the embodiments of the present application, the terms "first" and "second" are used only for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features.

[0065] To facilitate understanding, exemplary descriptions of concepts related to the present application are provided for reference.

[0066] 1) A feature vector is a one-dimensional or multi-dimensional data vector. A feature vector can be used to represent an object (or object). In fields such as AI and machine learning, a feature is an independent attribute or characteristic of an object. One feature is generally not enough to represent an object, so a combination of features, namely a feature vector, can be used to characterize an object. Therefore, a feature vector can be understood as a collection of attributes. A feature vector can usually be represented by a one-dimensional or multi-dimensional data vector. In a model training scenario, each sample has its own different attributes. Different attributes can be represented by different attribute values. Multiple attribute values ​​combined together can be represented by a vector, which is called a feature vector.

[0067] 2) Eigenvector table, which is a set of eigenvectors. The eigenvector table can be understood as a data table including m rows × n columns of data, where each data in the data table is a eigenvector, where m and n are both positive integers.

[0068] Each row of data in the feature vector table corresponds to a feature index, and the feature index is used to index the corresponding row of data. The feature index can be used to search and obtain data from the feature vector table.

[0069] It should be understood that in the embodiments of the present application, "at least one" refers to one or more, and "plurality" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the associated objects before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can represent: a, b, c, a and b, a and c, b and c, or a, b and c, where a, b, c can be single or multiple.

[0070] Recommendation models or recommendation networks based on recommendation models can be used to determine content that users may be interested in from massive amounts of content and recommend it to users. Currently, with the development of the Internet industry, the size of feature vectors in recommendation networks has increased exponentially, and the storage requirements of recommendation model training systems have also increased. At the same time, while storage requirements have increased, the computing power requirements of recommendation networks have not increased accordingly. Therefore, there is a scissors gap between computing power and storage, resulting in a waste of computing power for training equipment in recommendation model training systems. Therefore, heterogeneous storage of feature vectors is the main technology in the training system of recommendation models. Heterogeneous storage is to split feature vectors according to certain rules and store them on different device nodes. Through heterogeneous storage, it is possible to achieve a balance between computing power, storage, and communication in the recommendation model training system, thereby maximizing training efficiency.

[0071] Currently, the recommendation model training system usually consists of a master node and multiple device nodes. The master node can be used for training deployment and splitting the feature vector table containing feature vectors according to certain rules and storing them in multiple device nodes. Each of the multiple device nodes can be used to store the feature vector table data (i.e., feature vectors) assigned by the master node, and perform model training based on the training data. Among them, the training data is a collection of feature indexes, and a feature index can be used to index a row of feature vectors in the feature vector table. Therefore, the feature vector can also be understood as training data.

[0072] At present, the training data of the recommendation model is usually divided into n batches (bach), and the model training process of the recommendation model is usually divided into n training stages. Among them, n is a positive integer, and the n training stages correspond to the n batches of training data one by one. The training device can use the corresponding batch of training data to perform model training in each training stage of the n training stages. In the model training process of each stage, each device node can obtain the corresponding feature vector according to a batch of feature indexes required for the training process of this stage, and then complete the model training of this stage according to the obtained feature vector. Since multiple feature vector tables are split and stored in multiple device nodes according to certain rules, each device node needs to obtain the feature vector required for training from the device node and / or other device nodes. Among them, any two device nodes among the multiple device nodes can communicate directly, so each device node can directly send the feature index to other device nodes (i.e., all-to-all communication mode), and other device nodes can directly return the feature index to the device node after querying the feature vector corresponding to the feature index from the stored data.

[0073] The feature vector table segmentation methods (also called deployment methods) mainly include table segmentation method, row segmentation method, column segmentation method and data parallel method.

[0074] Among them, the table-wise method is to divide multiple feature vector tables according to the feature vector table type and store them in different device nodes. For example, the feature vector table corresponding to the user's age data can be stored in one device node, and the feature vector table corresponding to the user's height data can be stored in another device node, etc. Since the data volume of each feature vector table is different, in order to maintain the balance of data stored in the device node, the operation and maintenance personnel need to manually deploy when processing the feature vector table using the table-wise method. Therefore, this method needs to rely heavily on manual labor and is rarely used in actual applications.

[0075] The row splitting method is to divide each feature vector table into rows and store them in different device nodes. Different device nodes store different feature vectors, and each device node can store at least one row of feature vectors in the feature vector table. For example, when there are three device nodes, namely, device node 1, device node 2, and device node 3, if a feature vector table includes 10 rows of feature vectors, then the feature vectors in rows 1 to 3 of the feature vector table can be stored in device node 1, the feature vectors in rows 4 to 6 can be stored in device node 2, and the feature vectors in rows 7 to 10 can be stored in device node 3. The row splitting method can make the data in each feature vector table distributed on each device node, but the access to the feature vector table usually has a long tail effect, so some feature vectors may be accessed frequently, which will lead to unbalanced load between device nodes. For example, when the feature vectors in rows 7 to 10 in the above example are accessed frequently, the number of accesses to device node 3 is relatively large, which will lead to unbalanced access to device nodes 1 to 3, that is, unbalanced load on device nodes 1 to 3.

[0076] The column splitting method is to divide each feature vector table into columns and store them in different device nodes. Different device nodes store different feature vectors, and each device node can store at least one column of feature vectors in the feature vector table. For example, when there are three device nodes, namely, device node 1, device node 2, and device node 3, if a feature vector table includes 6 columns of feature vectors, then the feature vectors of columns 1 to 2 in the feature vector table can be stored in device node 1, the feature vectors of columns 3 to 4 can be stored in device node 2, and the feature vectors of columns 5 to 6 can be stored in device node 3. The column splitting method can make the data in each feature vector table distributed on each device node, and each device node will be accessed to obtain each row of data, which can ensure that the number of accesses to different device nodes is the same, thereby ensuring load balancing between devices. However, this method will cause the length of the feature vectors belonging to the same row stored in the same device node to become smaller. Each time a row of data is obtained, the device node needs to obtain part of the data from other device nodes with a smaller amount of access. For hardware, the efficiency of small packet access is low, so the data processing efficiency of this method is low.

[0077] The data parallel method is that each of the multiple device nodes stores a feature vector table, where a feature vector table can be one or more feature vector tables to be stored. In this method, when obtaining a feature vector, the device node can query the required feature vector from a feature vector table stored in itself, without having to obtain it from other device nodes, so the required communication volume is low and the efficiency is high. However, when the amount of data in the feature vector table is large, the storage capacity of the device node may not be enough to store a feature vector table, so this method is only suitable for processing feature vector tables with a small amount of data.

[0078] Currently, one can manually select a segmentation method from the above multiple segmentation methods for application, but manual selection is often limited by the experience of operation and maintenance personnel. Inexperienced operation and maintenance personnel may not be able to maximize the efficiency of training equipment. At the same time, a single segmentation method is often not the optimal solution for the training system. In order to solve the above problems, some current recommendation systems can use automatic parallelization to select the segmentation method. In the automatic parallelization method, the master node can automatically select the segmentation method according to the business scale and hardware parameters, and can use the automatically selected segmentation method to process the feature vector table during model training. However, this method only considers the total communication volume between device nodes during training, and does not consider the distribution law of data. In the recommendation model, data generally follows a power-law distribution, which will lead to an unbalanced communication load between device nodes. Therefore, the automatic parallel selection of the segmentation strategy is not the optimal solution for the training system, which will lead to a waste of computing power.

[0079] In summary, in practical applications, how to select a suitable segmentation method from a variety of segmentation methods to process the feature vector table, and then achieve a balance between the computing power, storage, and communication of the recommendation model training system to maximize the training efficiency, is still a problem to be solved.

[0080] In view of the above problems, in order to improve the selection efficiency of the feature vector table segmentation strategy, improve the balance of the device node load, and thus improve the efficiency of model training, the embodiment of the present application provides a data processing method and device, which can automatically select the segmentation method corresponding to each feature vector table in combination with the amount of data in the feature vector table to be stored and the data storage capacity of the device node, and can improve the selection efficiency of the feature vector table segmentation strategy. The method can be applied to a model training system, for example, it can be applied to a training system for a recommendation model. When the method is applied to a training system for a recommendation model, by selecting a segmentation method based on data storage requirements and data storage capacity, it is possible to coordinate the balance between storage capacity and storage requirements. The storage and communication load of the device node can be controlled through the selected segmentation method, which helps to eliminate the scissors gap between storage, communication, and computing power, thereby improving the efficiency of model training.

[0081] The scheme provided by the embodiment of the present application is described in detail below in conjunction with the accompanying drawings. The scheme provided by the embodiment of the present application can be executed by any computing device with processing and computing capabilities, and the computing device can be used as the main node in the model training system. The computing device can be an electronic device such as an AI training device or other device, which is not particularly limited in this application, as long as the program code corresponding to the technical solution provided in this application can be read and run to implement the data processing method provided in this application.

[0082] Figure 1 Schematic diagram of a scenario to which a data processing method provided in an embodiment of the present application is applicable. The solution provided in an embodiment of the present application can be applied in a model training system. Figure 1 As shown in , the model training system may include a master node and multiple device nodes. Among them, the master node is a server (or called an AI training cluster server), and the master node can determine the corresponding segmentation method for each feature vector table to be stored based on the amount of data in the feature vector table to be stored and the data storage capacity of the device node and other information, and split each feature vector table according to the corresponding segmentation method and store it in the device node. The device node is a device end (or called an AI training cluster device end), and the device node can store the data in the feature vector table according to the deployment of the master node, and perform model training based on the data in the feature vector table. Optionally, the master node and the device node can be AI training devices.

[0083] In some embodiments of the present application, the master node and each device node may be connected via a high-speed serial computer expansion bus standard (peripheral component interconnect express, PCIe) line.

[0084] In some embodiments of the present application, the master node may include: Figure 2 The first configuration information acquisition unit, the second configuration information acquisition unit, the automatic segmentation unit and the table deployment unit shown in .

[0085] The first configuration information acquisition unit may be used to acquire hardware configuration information of the device node, and the hardware configuration information may include at least one of the following: bandwidth between multiple device nodes, memory bandwidth of each device node, data volume of multiple device nodes, and memory cache line size of each device node. Optionally, the memory of the device node may be a high bandwidth memory (HBM).

[0086] Typically, the software and hardware configurations of multiple device nodes in the model training system are the same, so the hardware configuration information of each device node in the multiple device nodes can be the same. Figure 2 Taking the device nodes 1 to 3 shown in FIG as an example, the memory bandwidths of the device nodes 1 to 3 may be the same, the bandwidths between any two device nodes in the device nodes 1 to 3 may be the same, and the memory cache line sizes of the device nodes 1 to 3 may be the same.

[0087] The second configuration information acquisition unit can be used to acquire the business configuration information of the model training business, and the business configuration information may include at least one of the following: batch size, the number of feature vectors included in each training data (i.e., the number of feature vectors in a row corresponding to each feature index), the feature vector dimension, and the probability of the target data being stored in any device node. Optionally, the target data may be set data or a batch of training data corresponding to any training stage in the model training process.

[0088] Among them, the hardware configuration information and the business configuration information can be used to select the segmentation method corresponding to the feature vector table in the row segmentation method and the column segmentation method. The specific selection method can refer to the description in the following embodiment and will not be described in detail here.

[0089] The automatic segmentation unit can be used to determine the segmentation method corresponding to the feature vector table to be stored, and the table deployment unit can split the feature vector table and store it in multiple device nodes or adjust the deployment method of the feature vector table in multiple device nodes according to the segmentation method corresponding to the feature vector table determined by the automatic segmentation unit. The specific process can refer to the data processing method described in the embodiment below, which will not be described in detail here.

[0090] It should be noted that the application scenarios and system architectures described above are only examples. In actual applications, the model training system may include more or fewer nodes (or devices), or the functions of each node may be divided in other ways, or the node functions may not be divided, but the system works as a whole. The functional services in the master node described above are only examples. In actual applications, the master node may include more or fewer functional modules (or services), or the functions of each module may be divided in other ways, or the functional modules may not be divided, but the system works as a whole.

[0091] Based on the above Figure 1 and Figure 2 For the system architecture shown, please refer to Figure 3 , Figure 3 A data processing method provided in an embodiment of the present application is Figure 1 and Figure 2Based on the system architecture shown, the method may include:

[0092] S301: The master node obtains a plurality of feature vector tables to be stored; wherein each feature vector table in the plurality of feature vector tables includes one or more feature vectors.

[0093] In some embodiments of the present application, before acquiring multiple feature vector tables to be stored, the master node may determine the start time of a set period. That is, the data processing method provided in the embodiments of the present application may be executed periodically, and when the start time of each period is reached, the master node acquires multiple feature vector tables to be stored.

[0094] Optionally, the method provided in the embodiment of the present application can be executed during the AI ​​model training process. Since a training process of an AI model is usually completed in multiple stages, the execution cycle of the data processing method provided in the embodiment of the present application can be set to include at least one training stage. For example, a training process of an AI model may include N training stages, then the above-mentioned set cycle may include at least one training stage, and the number of the at least one training stage may be a positive integer less than or equal to N or a positive integer divisible by N. Based on this method, the master node can periodically select a segmentation strategy during the training process, and can adaptively adjust the segmentation strategy based on the selection result, thereby improving the utilization rate of device storage resources, and at the same time, it can basically not affect the model training process of the device node, and the efficiency is high. Among them, the adjustment of the segmentation strategy can be to adjust the deployment of multiple feature vector tables on multiple device nodes according to the determined segmentation strategy. In this process, the adjustment can be made at the granularity of the feature vector table, so the processing overhead is low and the resource occupancy of the model training system is low.

[0095] S302: The master node determines a first type of feature vector table and a second type of feature vector table among multiple feature vector tables; wherein the sum of the data storage capacity corresponding to the first type of feature vector table and the data storage capacity corresponding to the second type of feature vector table is less than or equal to the total data storage capacity of multiple device nodes, the data storage capacity corresponding to the first type of feature vector table is the product of the total data volume of the first type of feature vector table and the number of multiple device nodes, and the data storage capacity corresponding to the second type of feature vector table is the total data volume of the second type of feature vector table.

[0096] The first type of feature vector table includes at least one feature vector table among multiple feature vector tables, the second type of feature vector table includes at least one feature vector table among multiple feature vector tables, and the feature vector tables included in the first type of feature vector table are different from the feature vector tables included in the second type of feature vector table.

[0097] In an embodiment of the present application, multiple device nodes are used to store multiple feature vector tables. The first type of feature vector tables in the multiple feature vector tables belong to feature vector tables processed in a data parallel manner, and the data storage capacity corresponding to the first type of feature vector tables is the storage capacity required to store the first type of feature vector tables. Therefore, the data storage capacity corresponding to the first type of feature vector tables is the product of the total amount of data in the first type of feature vector tables and the number of multiple device nodes. The second type of feature vector tables in the multiple feature vector tables belong to feature vector tables processed in a row splitting manner and a column splitting manner, and the storage capacity corresponding to the second type of feature vector tables is the storage capacity required to store the second type of feature vector tables. Therefore, the data storage capacity corresponding to the second type of feature vector tables is the total amount of data in the second type of feature vector tables.

[0098] In some embodiments of the present application, in order to ensure that the data storage capacity of multiple device nodes is sufficient for processing the feature vector table according to the segmentation method corresponding to each feature vector table in the multiple feature vector tables, the master node can divide the multiple feature vector tables into the first category feature vector table and the second category feature vector table under the premise that "the sum of the data storage capacity corresponding to the first category feature vector table and the data storage capacity corresponding to the second category feature vector table is less than or equal to the total data storage capacity of the multiple device nodes". It can also be understood that the first category feature vector table and the second category feature vector table in the multiple feature vector tables meet the following conditions: the sum of the data storage capacity corresponding to the first category feature vector table and the data storage capacity corresponding to the second category feature vector table is less than or equal to the total data storage capacity of the multiple device nodes.

[0099] In some embodiments of the present application, the first type of feature vector table and the second type of feature vector table determined by the master node may further satisfy the following conditions while satisfying the above conditions: the data volume of the feature vector table with the smallest data volume in the second type of feature vector table is greater than or equal to the data volume of the feature vector table with the largest data volume in the first type of feature vector table. The first type of feature vector table determined based on the method includes at least one feature vector table with a smaller data volume in multiple vector tables, and the second type of feature vector table determined based on the method includes at least one feature vector table with a larger data volume in multiple vector tables. Since among the row segmentation method, the column segmentation method and the data parallel method, the data parallel method does not have the problem of load imbalance between device nodes and has the highest data access efficiency, the method can ensure that the feature vector table with a smaller data volume is processed in a data parallel method as much as possible, thereby increasing the number of feature vector tables that can be processed in a data parallel method, thereby maximizing the data access efficiency.

[0100] In some embodiments of the present application, the first type of feature vector table and the second type of feature vector table determined by the master node may further satisfy the following conditions when the above conditions are met: the product of the first data volume and the number of multiple device nodes is greater than the difference between the total data storage capacity of the multiple device nodes and the second data volume; wherein the first data volume is the data volume of the first feature vector table, the first feature vector table is the feature vector table with the smallest data volume in the second type of feature vector table, and the second data volume is the total data volume of other feature vector tables in the second type of feature vector table except the first feature vector table. Through this method, the number of feature vector tables processed in a data parallel manner can be maximized when the total data storage capacity required to store multiple feature vector tables does not exceed the data storage capacity of the device node, which can maximize data access efficiency and thus maximize model training efficiency. At the same time, this method can make full use of the storage space of multiple device nodes.

[0101] In a possible solution, the master node may determine the first type of feature vector table and the second type of feature vector table among the multiple feature vector tables according to the following method:

[0102] Repeat the following division process until the termination condition is met to obtain a second type of feature vector table, wherein the termination condition may be that the total amount of data in the reference table is less than or equal to the first reference data;

[0103] The division process includes: taking a feature vector table that is not divided into the second category feature vector table among multiple feature vector tables as a reference table; determining first reference data, wherein the first reference data is the difference between the total data storage capacity of multiple device nodes and a third data volume, and the third data volume is the total data volume of the feature vector tables that have been divided into the second category feature vector tables; when the product of the total data volume of the reference table and the data volume of multiple device nodes is greater than the first reference data, dividing the feature vector table with the largest data volume in the reference table into the second category feature vector table; dividing the feature vector table that is not divided into the second category feature vector table among multiple feature vector tables into the first category feature vector table.

[0104] Through the above-mentioned division process, the master node can select at least one feature vector table with the largest data volume from multiple feature vector tables, and divide the at least one feature vector table into the second category of feature vector tables, while the total data storage capacity required to store multiple feature vector tables does not exceed the total data storage capacity of multiple device nodes. Furthermore, the master node can select at least one feature vector table with the smallest data volume from multiple feature vector tables, and divide the at least one feature vector table into the first category of feature vector tables, thereby maximizing the number of feature vector tables processed in a data parallel manner.

[0105] In order to make it easier to understand the above division process, in order to determine the first type of feature vector table and the second type of feature vector table in multiple feature vector tables, the following will combine Figure 4 and one A specific example is used to describe the above division process in more detail.

[0106] As an optional implementation, Figure 4 As shown in , the master node can implement the above division process through the division method described in the following steps A1 to A7.

[0107] A1: The master node uses multiple feature vector tables as reference tables.

[0108] A2: The master node determines whether the product of the total data volume of the reference table and the number of multiple device nodes is greater than the total data storage capacity of the multiple device nodes; if so, execute step A3, otherwise, execute step A7.

[0109] The total data storage capacity of the multiple device nodes is the total data storage capacity available to the multiple device nodes at the beginning, which can be used to store data.

[0110] A3: The master node divides the feature vector table with the largest amount of data in the reference table into the second type of feature vector table; and executes step A4.

[0111] A4: The master node uses the feature vector table that is not classified as the second type of feature vector table among the multiple vector tables as a reference table; and executes step A5.

[0112] A5: The master node determines whether the product of the total data volume of the reference table and the number of multiple device nodes is greater than the difference between the total data storage capacity of the multiple device nodes and the total data volume of the second type feature vector table; if so, execute step A3, otherwise, execute step A6.

[0113] The difference between the total data storage capacity of the multiple device nodes and the total data volume of the second-type feature vector table is the total data storage capacity currently available to the multiple device nodes.

[0114] A6: The master node finishes the division, obtains the second type of feature vector table, and divides the feature vector tables that are not divided into the second type of feature vector tables among the multiple feature vector tables into the first type of feature vector tables.

[0115] If there is no feature vector table in the multiple feature vector tables that is not classified as the second type of feature vector table, then the multiple feature vector tables are all classified as the second type of feature vector tables.

[0116] A7: The master node divides the multiple feature vector tables into the first type of feature vector tables.

[0117] Combination Figure 4As described above, a specific example is given below to illustrate the implementation details of the above division process in more detail. It is assumed that the multiple feature vector tables to be stored may include feature vector tables 1 to 3, a total of 3 feature vector tables, wherein the data volume of feature vector table 1 is d1, the data volume of feature vector table 2 is d2, and the data volume of feature vector table 3 is d3, d1<d2<d3. Multiple device nodes may include Figure 2 There are a total of three device nodes, namely, device nodes 1 to 3 shown in , wherein the data storage capacity of each device node in device nodes 1 to 3 is d. Wherein, d1+d2+d3>d. In this scenario, based on the method provided in the above embodiment, when the master node classifies the three feature vector tables, it can first determine the product of the total data volume of the three feature vector tables (i.e., d1+d2+d3) and the number of three device nodes (i.e., 3) as 3(d1+d2+d3) according to the method described in the above steps A1 to A2, and this value is greater than the total data storage capacity of the three device nodes (i.e., 3d). Then the master node selects a feature vector table with the largest data volume, i.e., feature vector table 3, from the three feature vector tables according to the method described in the above step A3, and classifies feature vector table 3 as a second-category feature vector table. Then, the master node can use feature vector table 1 and feature vector table 2 as reference tables according to the method described in step A4 above, and determine according to the method described in step A5 above that the product of the total data volume of feature vector table 1 and feature vector table 2 (i.e., d1+d2) and the number of three device nodes (i.e., 3) is 3(d1+d2). Assuming that this value is less than the difference (i.e., 3d-d3) between the total data storage capacity of the three device nodes (i.e., 3d) and the total data volume of the second type of feature vector table (i.e., d3), it can be determined that the division process is over. After the division process is over, it can be determined that feature vector table 3 belongs to the second type of feature vector table, and feature vector table 1 and feature vector table 2 belong to the first type of feature vector table. Then, Figure 2 As shown in , the master node can store feature vector table 1 and feature vector table 2 in device nodes 1 to 3 respectively, and can split feature vector table 3 into three parts of data, table slices 1 to 3, and store table slices 1 to 3 in device nodes 1 to 3 respectively. The master node can split feature vector table 3 into table slices 1 to 3 in a row splitting manner or a column splitting manner.

[0118] S303: The master node stores the first type of feature vector table in each device node of the multiple device nodes respectively; and stores multiple feature vectors in at least one feature vector table included in the second type of feature vector table in different device nodes of the multiple device nodes, and the feature vectors stored in different device nodes are different.

[0119] Among them, after the master node stores the first-class feature vector table in each of the multiple device nodes, each device node stores a copy of the first-class feature vector table. When it is necessary to obtain the feature vector in the first-class feature vector table, each device node can query the required feature vector from the locally stored data without accessing other device nodes. Therefore, when the model is trained according to the feature vector in the first-class feature vector table, there is no communication load between the device nodes, which can improve the training efficiency.

[0120] In some embodiments of the present application, the way in which the master node stores multiple feature vectors in at least one feature vector table included in the second type of feature vector table to different device nodes among multiple device nodes may be: dividing the multiple feature vectors in at least one feature vector table included in the second type of feature vector table into different partial feature vectors; and storing the different partial feature vectors to different device nodes among the multiple device nodes.

[0121] Exemplarily, the above combination Figure 4 In the specific example listed, feature vector table 1 and feature vector table 2 belong to the first type of feature vector table, and feature vector table 3 belongs to the second type of feature vector table. Figure 2 As shown, through the above division and storage operations, the master node can store feature vector table 1, feature vector table 2 and table slice 1 in device node 1, store feature vector table 1, feature vector table 2 and table slice 2 in device node 2, and store feature vector table 1, feature vector table 2 and table slice 3 in device node 3; wherein, table slice 1, table slice 2 and table slice 3 are obtained by dividing multiple feature vectors in at least one feature vector table included in the second category of feature vector tables (i.e., multiple feature vectors in feature vector table 3), different table slices include part of the feature vectors in the second category of feature vector tables, and the feature vectors in each table slice are different.

[0122] In a possible solution, the partial feature vector may be at least one row of feature vectors or at least one column of feature vectors in at least one feature vector table included in the second type of feature vector table. When the partial feature vector is at least one row of feature vectors, the method is that the master node processes each feature vector table in the second type of feature vector table by using a row splitting method. When the partial feature vector is at least one column of feature vectors, the method is that the master node processes each feature vector table in the second type of feature vector table by using a column splitting method.

[0123] In another possible solution, the master node can determine the adopted segmentation method according to the corresponding scoring data when processing the feature vector table in different segmentation methods, and then determine the types of the above-mentioned partial feature vectors according to the determined segmentation method. Among them, the scoring data corresponding to the segmentation method is used to indicate the length of time required for the device node to obtain the feature vector in the feature vector table after processing a feature vector table according to the segmentation method. The size of the scoring data is positively correlated with the length of time required, that is, the larger the scoring data, the longer the time required, and the smaller the scoring data, the shorter the time required. The shorter the time required for the device node to obtain the feature vector in the feature vector table, the higher the efficiency, so the master node can use the segmentation method with the smallest scoring data as the adopted segmentation method.

[0124] In some embodiments of the present application, the above-mentioned different segmentation methods may include a row segmentation method and a column segmentation method. When the master node determines to adopt the row segmentation method, the above-mentioned partial feature vector is at least one row of feature vectors, and when the master node determines to adopt the column segmentation method, the above-mentioned partial feature vector is at least one column of feature vectors. Among them, the scoring data corresponding to the row segmentation method can be the first scoring data, and the scoring data corresponding to the column segmentation method can be the second scoring data. For each feature vector table in the second type of feature vector table, the master node can respectively determine the first scoring data corresponding to the row segmentation method and the second scoring data corresponding to the column segmentation method, and determine the segmentation method corresponding to the feature vector table based on the first scoring data and the second scoring data. For example, when the master node determines that the first scoring data is less than the second scoring data, the master node can determine that the partial feature vector is at least one row of feature vectors, that is, the feature vector table is processed by the row segmentation method. When the master node determines that the first scoring data is greater than the second scoring data, the master node can determine that the partial feature vector is at least one column of feature vectors, that is, the feature vector table is processed by the column segmentation method. When the master node determines that the first scoring data is equal to the second scoring data, the master node can determine that the partial feature vector is at least one row of feature vectors or at least one column of feature vectors, that is, the feature vector table can be processed by either row splitting or column splitting.

[0125] In some embodiments of the present application, the first scoring data can be specifically used to indicate: when part of the feature vector is at least one row of feature vectors, that is, when the feature vector table is processed by row splitting, the length of time required for the first device node to obtain the target feature vector; at this time, the target feature vector includes one or more rows of feature vectors in the second feature vector table, and the second feature vector table is any feature vector table in the second category of feature vector tables. The second scoring data can be specifically used to indicate: when part of the feature vector is at least one column of feature vectors, that is, when the feature vector table is processed by column splitting, the length of time required for the first device node to obtain the target feature vector. At this time, the target feature vector includes one or more columns of feature vectors in the second feature vector table, and the second feature vector table is any feature vector table in the second category of feature vector tables. The first device node can be Figure 2 Any of the multiple device nodes shown.

[0126] In some embodiments of the present application, the target feature vector may be any feature vector of N feature vectors; wherein the N feature vectors constitute a second feature vector table, and the second feature vector table is used to train the AI ​​model. The training process of the AI ​​model includes N training stages, and the N feature vectors correspond one-to-one to the N training stages. Each feature vector of the N feature vectors is used as training data for a corresponding training stage, and N is a positive integer. In the current AI model training method, the training data corresponding to a model training process is usually divided into multiple batches of data, and each batch of data can be used to complete a stage in a model training process. Therefore, each of the above-mentioned N pieces of data can be data belonging to the second feature vector table in a batch of data used in the corresponding training stage.

[0127] The method for calculating the first scoring data (ie, the scoring data corresponding to the row segmentation method) is described in detail below.

[0128] In the embodiment of the present application, the master node may obtain the first scoring data according to the methods described in the following steps B1 to B3.

[0129] B1: The master node obtains first time data; wherein the first time data is used to indicate the length of time required for the first device node to obtain the first data, and the first data includes: when the partial feature vector is at least one row of feature vectors, the feature vector stored in the first device node in the target feature vector.

[0130] After obtaining the data volume of the target data, the first probability data, and the first rate data, the master node can determine the data volume of the first data according to the data volume of the target data and the first probability data, and then determine the first time data according to the data volume of the first data and the first rate data. The first probability data is used to indicate the probability that the target data is stored in the first device node, and the first rate data is used to indicate the rate at which the first device node reads local data.

[0131] In this method, the master node may use the product of the amount of target data and the first probability data as the amount of first data. The ratio of the amount of first data to the first rate data is used as the first time data. In an optional implementation, the master node may calculate the first time data according to the amount of target data, the first probability data and the first rate data according to the following formula:

[0132]

[0133] Wherein, t1 is the first time data, data is the data volume of the target data, P is the first probability data, and BW1 is the first rate data. data*P is the estimated value, and data*P is used to represent the data volume of the first data (i.e., the data volume that the first device node needs to access when acquiring the target data, and can also be understood as the data volume that the first device node needs to acquire from the device node when acquiring the target data). Optionally, the value of BW1 can be the memory bandwidth of the first device node.

[0134] In some embodiments of the present application, during the initial training, it can be considered that the probability of storing the target data in each device node is the same, so the value of the first probability at the beginning can be the reciprocal of the number of device nodes. In the subsequent training process, the master node can determine the first probability data by counting the distribution law of the data used for training in the historical training process. Specifically, the master node can determine at least one device node for storing the first reference data by counting the distribution of the first reference data in the historical training process, and determine the probability that each device node in the at least one device node needs to access the data belonging to the first reference data from the local, and then the average value of the probability that the data belonging to the first reference data corresponding to the at least one device node needs to be accessed from the local can be used as the first probability data. Among them, the first reference data is the data belonging to the second feature vector table that needs to be obtained in the historical training process; the probability that each device node needs to access the data belonging to the first reference data from the local can be the ratio of the amount of data belonging to the first reference data stored by the device node to the amount of data in the second feature vector table. Among them, the historical training process can be at least one training stage included in the previous cycle, or can include at least one training stage before the current training stage. The process of the master node performing the above statistics can be executed in parallel with the process of the device node performing model training, and will not increase the additional performance loss of the device node.

[0135] The calculation process of the above-mentioned first probability data is described in more detail below with reference to a specific example. Assuming that the second feature vector table is feature vector table 1 and feature vector table 1 is stored in device nodes 1 to 5, assuming that in the historical training process, the data belonging to feature vector table 1 that needs to be obtained includes 85% of the data in feature vector table 1 (that is, the first reference data includes 85% of the data in feature vector table 1). Among them, 85% of the data in feature vector table 1 includes 20% of the data stored in device node 1, 30% of the data stored in device node 2, 25% of the data stored in device node 3, 9% of the data stored in device node 4, and 1% of the data stored in device node 5, then the first probability data P = (20% + 30% + 25% + 9% + 1%) / 5 = 0.17.

[0136] In some embodiments of the present application, the master node may calculate the data volume data of the target data according to the following formula:

[0137] data=BS*FC*DIM*B

[0138] Among them, BS is the number of feature indexes corresponding to the target data, FC is the number of feature vectors contained in a row of feature vectors corresponding to each feature index, DIM is the dimension of the feature vector, B is the data type of a data on a dimension of the feature vector, and B can be used to represent the maximum storage capacity of the storage space used to store the data in the device node. The value of B is a set value. Exemplarily, B can be a byte-level value such as 4 bytes.

[0139] In some embodiments of the present application, the master node can obtain the above-mentioned BS, BW1 according to the device hardware configuration information of the device node, and can obtain the above-mentioned FC, DIM, initial P and other data according to the service configuration information of the training service, and then calculate the first time data according to these data according to the above method.

[0140] B2: The master node obtains second time data, wherein the second time data is used to indicate the length of time required for the first device node to obtain the second data, and the second data includes: when the partial feature vector is at least one row of feature vectors, the feature vectors of device nodes other than the first device node are stored in the target feature vector.

[0141] The master node can obtain the data volume of the target data, the second probability data, and the second rate data, and then determine the data volume of the second data according to the data volume of the target data and the second probability data, and then determine the second time data according to the data volume of the second data and the second rate data; wherein the second probability data is used to indicate the probability that the target data is stored in a device node other than the first device node, and the second rate data is used to indicate the data transmission rate between the first device node and the device nodes other than the first device node.

[0142] In this method, the master node can use the product of the amount of target data and the second probability data as the amount of second data. The ratio of the amount of second data to the second rate data is used as the first time data. In an optional implementation, the master node can calculate the second time data according to the amount of target data, the second probability data and the second rate data according to the following formula:

[0143]

[0144] Wherein, t2 is the second time data, data is the data volume of the target data, p is the second probability data, and BW2 is the data transmission rate between the first device node and the device nodes other than the first device node. data*p is an estimated value, and data*p is used to represent the data volume of the second data (i.e., the data volume of other device nodes other than the present device node that the first device node needs to access when acquiring the target data, which can also be understood as the data volume that the first device node needs to acquire from other device nodes other than the present device node when acquiring the target data). The value of p2 is 1-P. The calculation method of data can refer to the method described in the above step B1, which will not be repeated here.

[0145] In some embodiments of the present application, the master node can obtain the above-mentioned BS, BW2 according to the device hardware configuration information of the device node, and can obtain the above-mentioned FC, DIM, initial p and other data according to the service configuration information of the training service, and then calculate the second time data according to these data according to the above method.

[0146] Optionally, the execution order of the above steps B1 and B2 can be arbitrary, and is not limited in the embodiments of the present application. For example, steps B1 and B2 can be executed simultaneously, or any one of steps B1 and B2 can be executed first and the other one can be executed later.

[0147] B3: The master node determines first scoring data according to the first time data and the second time data.

[0148] In some embodiments of the present application, the master node may use the value obtained by weighted summing the first time data and the second time data as the first scoring data. The weight corresponding to the first time data and the second time data may be a set value. The calculation formula of the first scoring data may conform to the following formula:

[0149]

[0150] PERF1 is the first scoring data. Regarding BS, FC, DIM, P, BW1, and BW2, reference may be made to the corresponding descriptions in the above embodiments, which will not be repeated here.

[0151] The method for calculating the second scoring data (ie, the scoring data corresponding to the column segmentation method) is described in detail below.

[0152] In the embodiment of the present application, the master node may obtain the first scoring data according to the methods described in the following steps C1 to C3.

[0153] C1: The master node obtains third time data; wherein the third time data is used to indicate the length of time required for the first device node to obtain the third data, and the third data includes: when the partial feature vector is at least one column of feature vectors, the feature vector stored in the first device node in the target feature vector.

[0154] The master node can obtain the data volume of the target data, the number of multiple device nodes, the first rate data, and the set first coefficient, and then determine the data volume of the third data based on the data volume of the target data and the number of multiple device nodes, and then determine the third time data based on the data volume of the third data, the first rate data and the first coefficient; wherein the first rate data is used to indicate the rate at which the first device node reads local data.

[0155] In some embodiments of the present application, the master node may calculate the data volume of the third data according to the following formula:

[0156]

[0157] Among them, data3 is the data volume of the third data, BS is the number of feature indexes corresponding to the target data, FC is the number of feature vectors contained in a row of feature vectors corresponding to each feature index, DIM is the dimension of the feature vector, B is the data type of a data on one dimension of the feature vector, and K is the number of multiple device nodes.

[0158] In an optional implementation manner, the master node may calculate the third time data according to the data volume of the third data, the first rate data, and the set first coefficient according to the following formula:

[0159]

[0160] Among them, t3 is the third time data, data3 is the data volume of the third data (that is, the data volume that the first device node needs to access when acquiring the target data, which can also be understood as the data volume that the first device node needs to acquire from the device node when acquiring the target data), C is the first coefficient (also called the column penalty coefficient), and BW1 is the first rate data. Optionally, the value of BW1 can be the memory bandwidth of the first device node.

[0161] Optionally, the first coefficient may be a penalty coefficient of the column segmentation method, the first coefficient may be determined according to a hardware parameter of the device node, or the first coefficient may be a set value.

[0162] In one possible method, the first coefficient can be determined according to the following formula:

[0163] C=DIM*B / CLS

[0164] Wherein, C is the first coefficient, DIM is the dimension of the feature vector, B is the data type of a data on one dimension of the feature vector, and CLS is the memory cache line size.

[0165] Optionally, in actual application, the size of the first coefficient may also be determined through actual testing on the device node.

[0166] In some embodiments of the present application, the master node can obtain the above-mentioned BS, BW1, C according to the device hardware configuration information of the device node, and can obtain the above-mentioned FC, DIM and other data according to the service configuration information of the training service, and then calculate the third time data according to these data according to the above method.

[0167] C2: The master node obtains fourth time data; wherein the fourth time data is used to indicate the length of time required for the first device node to obtain the fourth data, and the fourth data includes: when the partial feature vector is at least one column of feature vectors, the feature vectors of device nodes other than the first device node are stored in the target feature vector.

[0168] The master node can obtain the data volume of the target data, the number of multiple device nodes, the second rate data, and the first coefficient, and then determine the data volume of the fourth data according to the data volume of the target data and the number of multiple device nodes, and then determine the fourth time data according to the data volume of the fourth data, the second rate data and the first coefficient; wherein the second rate data is used to indicate the data transmission rate between the first device node and device nodes other than the first device node.

[0169] In some embodiments of the present application, the master node may calculate the data volume of the fourth data according to the following formula:

[0170]

[0171] Among them, data4 is the data volume of the fourth data, BS is the number of feature indexes corresponding to the target data, FC is the number of feature vectors contained in a row of feature vectors corresponding to each feature index, DIM is the dimension of the feature vector, B is the data type of a data on one dimension of the feature vector, and K is the number of multiple device nodes.

[0172] In an optional implementation manner, the master node may calculate the fourth time data according to the data volume of the fourth data, the first rate data, and the set first coefficient according to the following formula:

[0173]

[0174] Among them, t4 is the third time data, data4 is the data volume of the fourth data (i.e., the data volume of other device nodes other than the first device node that the first device node needs to access when acquiring the target data, which can also be understood as the data volume that the first device node needs to acquire from other device nodes other than the first device node when acquiring the target data), C is the first coefficient, and BW2 is the data transmission rate between the first device node and the device nodes other than the first device node. Optionally, BW2 can be the bandwidth between multiple device nodes.

[0175] In some embodiments of the present application, the master node can obtain the above-mentioned BS, BW2, C according to the device hardware configuration information of the device node, and can obtain the above-mentioned FC, DIM and other data according to the service configuration information of the training service, and then calculate the third time data according to these data according to the above method.

[0176] Optionally, the execution order of the above steps C1 and C2 can be arbitrary and is not limited in the embodiments of the present application. For example, steps C1 and C2 can be executed simultaneously, or any one of steps C1 and C2 can be executed first and the other one can be executed later.

[0177] C3: The master node determines the second scoring data according to the third time data and the fourth time data.

[0178] In some embodiments of the present application, the master node may use the value obtained by weighted summing the third time data and the fourth time data as the second scoring data. The weights corresponding to the third time data and the fourth time data may be set values. The calculation formula of the second scoring data may conform to the following formula:

[0179]

[0180] PERF2 is the second scoring data. Regarding BS, FC, DIM, BW1, BW2, K, and C, reference may be made to the corresponding descriptions in the above embodiments, which will not be repeated here.

[0181] For example, in the above Figure 2 , Figure 3 and Figure 4In the embodiment shown, it can be seen from the specific example that the feature vector table 3 is divided into the second type of feature vector table. When it is split and stored in the device nodes 1 to 3, combined with the above specific description of the segmentation method, the master node can further select the segmentation method corresponding to the feature vector table 3 in the row segmentation method and the column segmentation method. Specifically, the master node can calculate the first scoring data corresponding to the feature vector table 3 according to the method described in the above steps B1 to B3, and the master node can calculate the second scoring data corresponding to the feature vector table 3 according to the method described in the above steps C1 to C3. When it is determined that the first scoring data is less than the second scoring data, the master node can process the feature vector table 3 in a row segmentation method to obtain multiple table slices. When it is determined that the first scoring data is greater than the second scoring data, the master node can process the feature vector table 3 in a column segmentation method to obtain multiple table slices. When it is determined that the first scoring data is equal to the second scoring data, the master node can process the feature vector table 3 in any one of the row segmentation method and the column segmentation method to obtain multiple table slices.

[0182] In the method provided in the above embodiment, the master node can determine the first type of feature vector table and the second type of feature vector table in multiple feature vector tables according to the data volume of the feature vector table and the data storage capacity of the device node, and process the first type of feature vector table in multiple feature vector tables in a data parallel manner. Optionally, the second type of feature vector table in multiple feature vector tables can be processed by row segmentation or column segmentation. Among them, the selection and corresponding processing of the segmentation method based on the data volume of the feature vector table and the data storage capacity of the device node can make more full and effective use of the storage space of the device node, thereby reducing or eliminating the scissors difference between the computing power and storage of the model training system, achieving a relative balance between the computing power and storage of the training system, and maintaining a relative balance between the data storage demand and the data storage space. By processing the first type of feature vector table in a data parallel manner, there is no communication load between device nodes when model training is performed according to the feature vectors in the first type of feature vector table, which can alleviate the problem of unbalanced communication load between device nodes and improve the model training efficiency. By processing the second type of feature vector table in multiple feature vector tables in a row segmentation or column segmentation manner, a certain amount of storage space can be saved.

[0183] Based on the solution provided in the above embodiments, an embodiment of the present application also provides a data processing method, which includes: a master node obtains multiple feature vector tables to be stored; wherein each feature vector table in the multiple feature vector tables includes one or more feature vectors; when the master node determines that the product of the total data volume of the multiple feature vector tables and the number of multiple device nodes is less than or equal to the total data storage capacity of the multiple device nodes, the multiple feature vector tables are stored in each device node in the multiple device nodes respectively.

[0184] Among them, after the master node stores multiple feature vector tables in each of the multiple device nodes, each device node stores a copy of the data of multiple feature vector tables. Then, during the model training process, each device node can query the feature vectors required for model training from the locally stored data without accessing other device nodes. Therefore, during the model training process, there is no communication load between device nodes, which can minimize the communication load between device nodes and improve training efficiency.

[0185] In some embodiments of the present application, the two data processing methods provided in the above embodiments can be executed in combination. The following is a specific example to illustrate the specific implementation process when the two data processing methods provided in the above embodiments are executed in combination, so as to more specifically and thoroughly illustrate the process of combining the two data processing methods provided in the above embodiments. Figure 5 A possible execution flow when the two data processing methods provided in the above embodiments are combined and executed may include:

[0186] S501: The master node obtains multiple feature vector tables to be stored.

[0187] S502: The master node obtains hardware configuration information of multiple device nodes and business configuration information of the model training business.

[0188] S503: The master node uses the multiple feature vector tables as reference tables.

[0189] S504: The master node determines the data storage capacity required for processing the reference table in a data parallel manner according to the data amount of the reference table.

[0190] S505: The master node determines whether the required data storage capacity is greater than the total data storage capacity of multiple device nodes; if so, execute step S506, otherwise, execute step S509.

[0191] S506: The master node selects a feature vector table with the largest amount of data from the reference table as a target feature vector table.

[0192] S507: The master node calculates the first scoring data and the second scoring data corresponding to the target feature vector table based on the acquired hardware configuration information and business configuration information, and selects a segmentation method corresponding to the target feature vector table from the row segmentation method and the column segmentation method based on the first scoring data and the second scoring data corresponding to the target feature vector table.

[0193] S508: The master node removes the target feature vector table from the reference table and executes step S504.

[0194] S509: The master node determines that the segmentation method corresponding to each feature vector table in the reference table is a data parallel method.

[0195] S510: The master node splits the multiple feature vector tables and stores them into multiple device nodes according to the segmentation method corresponding to each feature vector table in the multiple feature vector tables.

[0196] Optionally, the process may further include the following steps:

[0197] S511: After a fixed training phase interval, the master node re-determines the segmentation method corresponding to each vector table in the plurality of feature vector tables.

[0198] Wherein, during the process of the device node performing model training according to the feature vectors in multiple feature vector tables, the master node can count the probability data (including the first probability data and the second probability data described in the aforementioned embodiment) corresponding to the feature vector table processed in a row splitting manner. Wherein, the probability data corresponding to any feature vector table is used to indicate the probability that the data belonging to the feature vector table obtained during the device node training process is stored in the device node. During the process of re-determining the splitting method corresponding to each vector table in multiple feature vector tables, the master node can determine the first scoring data corresponding to the feature vector table based on the probability data corresponding to the feature vector table counted, so as to select a row splitting method or a column splitting method for the feature vector table.

[0199] S512: The master node adjusts the deployment of the multiple feature vector tables on the multiple device nodes according to the re-determined segmentation method.

[0200] The method described in steps S511 to S512 may be repeatedly executed until the model training process is completed.

[0201] In some embodiments of the present application, the above process can be performed by Figure 2 The first configuration information acquisition unit, the second configuration information acquisition unit, the automatic segmentation unit, and the table deployment unit shown in the figure are completed in collaboration. For example, the method described in step S501, steps S503 to S509, and step S511 can be performed by the automatic segmentation unit, the method for acquiring hardware information in step S502 can be performed by the first configuration information acquisition unit, the method for acquiring business configuration information in step S502 can be completed by the second configuration information acquisition unit, and the methods described in steps S510 and S512 can be completed by the table deployment unit.

[0202] The specific implementation of the terms, concepts and methods mentioned in the above process can all refer to the corresponding descriptions in the above embodiments, and will not be described in detail here.

[0203] It should be noted that the specific implementation processes provided in the above embodiments are only examples of the applicable method processes of the embodiments of the present application, wherein the execution order of each step can be adjusted accordingly according to actual needs, and other steps can be added or some steps can be reduced. The specific implementation methods of each step can refer to the relevant introduction in the previous embodiments, and will not be described in detail here.

[0204] Based on the above embodiments and the same technical concept, the embodiments of the present application also provide a data processing device, which can be used to implement the functions of the master node provided in the embodiments of the present application. Figure 6 The figure shows a data processing device 600 provided in an embodiment of the present application. Optionally, the data processing device 600 may be a master node device, or a chip or chip system or integrated circuit in the master node device. Figure 6 As shown in , the data processing device 600 may include a data acquisition unit 601 and a data processing unit 602, wherein the data acquisition unit 601 cooperates with the data processing unit 602 to execute the method performed by the master node provided in an embodiment of the present application.

[0205] The division of functional units (or functional modules) in the embodiments of the present application is schematic and is only a logical functional division. There may be other division methods in actual implementation. In addition, each functional unit in each embodiment of the present application may be integrated into a processor, or may exist physically separately, or two or more functional units may be integrated into one module. The above-mentioned integrated functional units may be implemented in the form of hardware or in the form of software functional modules.

[0206] Figure 6 One or more of the various units in the system may be implemented by software, hardware, firmware or a combination thereof. The software or firmware includes but is not limited to computer program instructions or codes, and may be executed by a hardware processor. The hardware includes but is not limited to various integrated circuits, such as a central processing unit (CPU), a digital signal processor (DSP), a field programmable gate array (FPGA) or an application specific integrated circuit (ASIC).

[0207] Based on the above embodiments and the same technical concept, the embodiments of the present application also provide a data processing device, which can be used to implement the data processing method provided in the embodiments of the present application. Figure 7 The figure shows a data processing device 700 provided in an embodiment of the present application. Optionally, the data processing device 700 may be a master node device, or a chip or chip system or integrated circuit in the master node device. Figure 7As shown in the figure, the data processing device 700 may include: one or more processors 701, a memory 702, and one or more computer programs (not shown in the figure). As an implementation, the above-mentioned devices can be coupled via one or more communication lines 703. Among them, one or more computer programs are stored in the memory 702, and the one or more computer programs include instructions; the processor 701 calls the instructions stored in the memory 702, so that the data processing device 700 performs the data processing method provided in the embodiment of the present application.

[0208] In the embodiments of the present application, the processor may be a general-purpose processor, a digital signal processor, an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component, and may implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of the present application. The general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application may be directly embodied as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor.

[0209] In the embodiment of the present application, the memory may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM) and direct RAM bus RAM (DR RAM). It should be noted that the memory of the system and method described herein is intended to include but is not limited to these and any other suitable types of memory. The memory in the embodiments of the present application can also be a circuit or any other device capable of realizing a storage function.

[0210] As an implementation method, the data processing device 700 may also include a communication interface 704 for communicating with other devices through a transmission medium. For example, the data processing device 700 may exchange information or data with a first server, a second server, or a database through the communication interface 704. In an embodiment of the present application, the communication interface may be a transceiver, a circuit, a bus, a module, or other types of communication interfaces. In an embodiment of the present application, when the communication interface is a transceiver, the transceiver may include an independent receiver, an independent transmitter; or a transceiver with integrated transceiver functions, or an interface circuit.

[0211] In some embodiments of the present application, the processor 701, the memory 702 and the communication interface 704 can be connected to each other via a communication line 703; the communication line 703 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The communication line 703 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 7 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0212] In the method provided in the embodiment of the present application, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, a network device, a user device or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from one website, computer, server or data center to another website, computer, server or data center via wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more available media integrated therein. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium (e.g., a digital video disc (DVD), or a semiconductor medium (e.g., an SSD), etc.

[0213] In one example, when the data processing device 700 is implemented as the master node in the above embodiment, the data processing device can be used to execute the method performed by the master node provided in the embodiment of the present application.

[0214] Based on the above embodiments and the same technical concept, an embodiment of the present application also provides a data processing device, which includes at least one processor and an interface; the interface is used to provide program instructions or data to the at least one processor; the at least one processor is used to execute the program instructions to implement the data processing method provided in the above embodiments of the present application.

[0215] Based on the above embodiments and the same technical concept, an embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program runs on a data processing device, the data processing device executes the data processing method provided in the above embodiments of the present application.

[0216] Based on the above embodiments and the same technical concept, the embodiments of the present application also provide a computer program product, which includes a computer program or instructions. When the computer program or instructions are executed by a data processing device, the data processing method provided in the above embodiments of the present application is implemented.

[0217] Based on the above embodiments and the same technical concept, an embodiment of the present application also provides a chip system, which includes at least one processor and an interface, wherein the interface is used to provide program instructions or data to the at least one processor, and the at least one processor is used to execute the program instructions to implement the data processing method provided in the above embodiments of the present application.

[0218] In one possible design, the chip system also includes a memory, which is used to store program instructions and data.

[0219] In one possible design, the chip system is composed of a chip, or includes a chip and other discrete devices.

[0220] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A data processing method, characterized in that: include: Acquire a plurality of feature vector tables to be stored; wherein each feature vector table in the plurality of feature vector tables includes one or more feature vectors; Determine a first type of feature vector table and a second type of feature vector table among the multiple feature vector tables; wherein the sum of the data storage capacity corresponding to the first type of feature vector table and the data storage capacity corresponding to the second type of feature vector table is less than or equal to the total data storage capacity of the multiple device nodes, the data storage capacity corresponding to the first type of feature vector table is the product of the total data volume of the first type of feature vector table and the number of the multiple device nodes, and the data storage capacity corresponding to the second type of feature vector table is the total data volume of the second type of feature vector table; storing the first type of feature vector table in each device node of the plurality of device nodes respectively; and A plurality of feature vectors in at least one feature vector table included in the second-type feature vector table are stored in different device nodes among the plurality of device nodes, and the feature vectors stored in different device nodes are different.

2. The method according to claim 1, characterized in that The data volume of the feature vector table with the smallest data volume in the second type of feature vector tables is greater than or equal to the data volume of the feature vector table with the largest data volume in the first type of feature vector tables.

3. The method according to claim 2, characterized in that The product of the first data volume and the number of the multiple device nodes is greater than the difference between the total data storage capacity of the multiple device nodes and the second data volume; wherein the first data volume is the data volume of the first feature vector table, the first feature vector table is the feature vector table with the smallest data volume in the second type of feature vector table, and the second data volume is the total data volume of other feature vector tables in the second type of feature vector table except the first feature vector table.

4. The method according to any one of claims 1 to 3, characterized in that: The determining of the first type of feature vector table and the second type of feature vector table among the plurality of feature vector tables comprises: Repeat the division process until the termination condition is met to obtain a second type of feature vector table; the division process includes: taking a feature vector table among the plurality of feature vector tables that is not classified as the second type of feature vector table as a reference table; Determine first reference data, wherein the first reference data is a difference between a total data storage capacity of the plurality of device nodes and a third data volume, and the third data volume is a total data volume of a feature vector table that has been classified as the second type of feature vector table; When the total data volume of the reference table is greater than the first reference data, classifying the feature vector table with the largest data volume in the reference table as the second type of feature vector table; The termination condition is that the total amount of data in the reference table is less than or equal to the first reference data; The feature vector tables among the multiple feature vector tables that are not classified as the second type of feature vector tables are classified as the first type of feature vector tables.

5. The method according to any one of claims 1 to 4, characterized in that: Storing a plurality of feature vectors in at least one feature vector table included in the second type of feature vector table to different device nodes among the plurality of device nodes comprises: Dividing a plurality of feature vectors in at least one feature vector table included in the second type of feature vector table into different partial feature vectors; The different partial feature vectors are stored in different device nodes among the plurality of device nodes.

6. The method according to claim 5, characterized in that When the first score data is less than the second score data, the partial feature vector is at least one row of feature vectors; When the first scoring data is greater than the second scoring data, the partial feature vector is at least one column of feature vectors; When the first scoring data is equal to the second scoring data, the partial feature vector is at least one row of feature vectors or at least one column of feature vectors; Among them, the first scoring data is used to indicate: when the partial feature vector is at least one row of feature vector, the length of time required for the first device node to obtain the target feature vector; the second scoring data is used to indicate: when the partial feature vector is at least one column of feature vector, the length of time required for the first device node to obtain the target feature vector; the target feature vector includes one or more rows of feature vectors in a second feature vector table, the second feature vector table is any feature vector table in the second category of feature vector tables, and the first device node is any device node among the multiple device nodes.

7. The method according to claim 6, characterized in that The method further comprises: Acquire first time data; wherein the first time data is used to indicate the length of time required for the first device node to acquire the first data, and the first data includes: when the partial feature vector is at least one row of feature vectors, the feature vector stored in the first device node in the target feature vector; Acquire second time data; wherein the second time data is used to indicate the length of time required for the first device node to acquire the second data, and the second data includes: when the partial feature vector is at least one row of feature vectors, feature vectors of device nodes other than the first device node stored in the target feature vector; The first scoring data is determined according to the first time data and the second time data.

8. The method according to claim 7, characterized in that The obtaining of the first time data includes: Acquire the data volume of the target feature vector, first probability data, and first rate data; wherein the first probability data is used to indicate the probability that the target feature vector is stored in the first device node, and the first rate data is used to indicate the rate at which the first device node reads local data; Determining the data volume of the first data according to the data volume of the target feature vector and the first probability data; The first time data is determined according to the data amount of the first data and the first rate data.

9. The method according to claim 7 or 8, characterized in that The obtaining of the second time data comprises: Acquire the data volume of the target feature vector, second probability data, and second rate data; wherein the second probability data is used to indicate the probability that the target feature vector is stored in a device node other than the first device node, and the second rate data is used to indicate the data transmission rate between the first device node and the device nodes other than the first device node; Determining the amount of the second data according to the number of the target feature vectors and the second probability data; The second time data is determined according to the data amount of the second data and the second rate data.

10. The method according to any one of claims 6 to 9, characterized in that: The method further comprises: Acquire third time data; wherein the third time data is used to indicate the length of time required for the first device node to acquire the third data, and the third data includes: when the partial feature vector is at least one column of feature vectors, the feature vector stored in the first device node in the target feature vector; Acquire fourth time data; wherein the fourth time data is used to indicate the length of time required for the first device node to acquire fourth data, and the fourth data includes: when the partial feature vector is at least one column of feature vectors, feature vectors of device nodes other than the first device node stored in the target feature vector; The second scoring data is determined according to the third time data and the fourth time data.

11. The method according to claim 10, characterized in that The obtaining of the third time data comprises: Acquire the data volume of the target feature vector, the number of the plurality of device nodes, first rate data, and a set first coefficient; wherein the first rate data is used to indicate the rate at which the first device node reads local data; Determining the data volume of the third data according to the data volume of the target feature vector and the number of the plurality of device nodes; The third time data is determined according to the data amount of the third data, the first rate data and the first coefficient.

12. The method according to claim 10, characterized in that The obtaining of the fourth time data comprises: Acquire the data volume of the target feature vector, the number of the plurality of device nodes, second rate data, and the first coefficient; wherein the second rate data is used to indicate the data transmission rate between the first device node and device nodes other than the first device node; Determining the data volume of the fourth data according to the data volume of the target feature vector and the number of the plurality of device nodes; The fourth time data is determined according to the data amount of the fourth data, the second rate data and the first coefficient.

13. The method according to any one of claims 6 to 12, characterized in that: The target feature vector is any one of the N feature vectors; wherein the N feature vectors constitute the second feature vector table, the second feature vector table is used to train the AI ​​model, the training process of the AI ​​model includes N training stages, the N feature vectors correspond one-to-one to the N training stages, each of the N feature vectors is used as training data for a corresponding training stage, and N is a positive integer.

14. The method according to claim 13, characterized in that Before obtaining a plurality of feature vector tables to be stored, the method further includes: Determine the start time of a set period; wherein the set period includes at least one of the training stages.

15. A data processing device, characterized in that: include: A data acquisition unit, used to acquire a plurality of feature vector tables to be stored; wherein each feature vector table in the plurality of feature vector tables includes one or more feature vectors; A data processing unit, used to determine a first type of feature vector table and a second type of feature vector table among the multiple feature vector tables; wherein the sum of the data storage capacity corresponding to the first type of feature vector table and the data storage capacity corresponding to the second type of feature vector table is less than or equal to the total data storage capacity of the multiple device nodes, the data storage capacity corresponding to the first type of feature vector table is the product of the total data volume of the first type of feature vector table and the number of the multiple device nodes, and the data storage capacity corresponding to the second type of feature vector table is the total data volume of the second type of feature vector table; The data processing unit is also used to store the first type of feature vector table in each device node of the multiple device nodes respectively; and to store multiple feature vectors in at least one feature vector table included in the second type of feature vector table in different device nodes of the multiple device nodes, and the feature vectors stored in different device nodes are different.

16. The data processing device according to claim 15, characterized in that The data volume of the feature vector table with the smallest data volume in the second type of feature vector tables is greater than or equal to the data volume of the feature vector table with the largest data volume in the first type of feature vector tables.

17. The data processing device according to claim 16, characterized in that: The product of the first data volume and the number of the multiple device nodes is greater than the difference between the total data storage capacity of the multiple device nodes and the second data volume; wherein the first data volume is the data volume of the first feature vector table, the first feature vector table is the feature vector table with the smallest data volume in the second type of feature vector table, and the second data volume is the total data volume of other feature vector tables in the second type of feature vector table except the first feature vector table.

18. The data processing device according to any one of claims 15 to 17, characterized in that: When the data processing unit determines the first type of feature vector table and the second type of feature vector table in the plurality of feature vector tables, it is specifically used to: Repeat the division process until the termination condition is met to obtain a second type of feature vector table; the division process includes: taking a feature vector table among the plurality of feature vector tables that is not classified as the second type of feature vector table as a reference table; Determine first reference data, wherein the first reference data is a difference between a total data storage capacity of the plurality of device nodes and a third data volume, and the third data volume is a total data volume of a feature vector table that has been classified as the second type of feature vector table; When the total data volume of the reference table is greater than the first reference data, classifying the feature vector table with the largest data volume in the reference table as the second type of feature vector table; The termination condition is that the total amount of data in the reference table is less than or equal to the first reference data; The feature vector tables among the multiple feature vector tables that are not classified as the second type of feature vector tables are classified as the first type of feature vector tables.

19. The data processing device according to any one of claims 15 to 18, characterized in that: When the data processing unit stores a plurality of feature vectors in at least one feature vector table included in the second type of feature vector table to different device nodes among the plurality of device nodes, it is specifically used to: Dividing a plurality of feature vectors in at least one feature vector table included in the second type of feature vector table into different partial feature vectors; The different partial feature vectors are stored in different device nodes among the plurality of device nodes.

20. The data processing device according to claim 19, characterized in that When the first score data is less than the second score data, the partial feature vector is at least one row of feature vectors; When the first scoring data is greater than the second scoring data, the partial feature vector is at least one column of feature vectors; When the first scoring data is equal to the second scoring data, the partial feature vector is at least one row of feature vectors or at least one column of feature vectors; Among them, the first scoring data is used to indicate: when the partial feature vector is at least one row of feature vector, the length of time required for the first device node to obtain the target feature vector; the second scoring data is used to indicate: when the partial feature vector is at least one column of feature vector, the length of time required for the first device node to obtain the target feature vector; the target feature vector includes one or more rows of feature vectors in a second feature vector table, the second feature vector table is any feature vector table in the second category of feature vector tables, and the first device node is any device node among the multiple device nodes.

21. The data processing device according to claim 20, characterized in that The data processing unit is also used for: Acquire first time data; wherein the first time data is used to indicate the length of time required for the first device node to acquire the first data, and the first data includes: when the partial feature vector is at least one row of feature vectors, the feature vector stored in the first device node in the target feature vector; Acquire second time data; wherein the second time data is used to indicate the length of time required for the first device node to acquire the second data, and the second data includes: when the partial feature vector is at least one row of feature vectors, feature vectors of device nodes other than the first device node stored in the target feature vector; The first scoring data is determined according to the first time data and the second time data.

22. The data processing device according to claim 21, characterized in that When the data processing unit acquires the first time data, it is specifically used to: Acquire the data volume of the target feature vector, first probability data, and first rate data; wherein the first probability data is used to indicate the probability that the target feature vector is stored in the first device node, and the first rate data is used to indicate the rate at which the first device node reads local data; Determining the data volume of the first data according to the data volume of the target feature vector and the first probability data; The first time data is determined according to the data amount of the first data and the first rate data.

23. The data processing device according to claim 21 or 22, characterized in that: When the data processing unit acquires the second time data, it is specifically used to: Acquire the data volume of the target feature vector, second probability data, and second rate data; wherein the second probability data is used to indicate the probability that the target feature vector is stored in a device node other than the first device node, and the second rate data is used to indicate the data transmission rate between the first device node and the device nodes other than the first device node; Determining the amount of the second data according to the number of the target feature vectors and the second probability data; The second time data is determined according to the data amount of the second data and the second rate data.

24. The data processing device according to any one of claims 20 to 23, characterized in that: The data processing unit is also used for: Acquire third time data; wherein the third time data is used to indicate the length of time required for the first device node to acquire the third data, and the third data includes: when the partial feature vector is at least one column of feature vectors, the feature vector stored in the first device node in the target feature vector; Acquire fourth time data; wherein the fourth time data is used to indicate the length of time required for the first device node to acquire fourth data, and the fourth data includes: when the partial feature vector is at least one column of feature vectors, feature vectors of device nodes other than the first device node stored in the target feature vector; The second scoring data is determined according to the third time data and the fourth time data.

25. The data processing device according to claim 24, characterized in that When the data processing unit acquires the third time data, it is specifically used to: Acquire the data volume of the target feature vector, the number of the plurality of device nodes, first rate data, and a set first coefficient; wherein the first rate data is used to indicate the rate at which the first device node reads local data; Determining the data volume of the third data according to the data volume of the target feature vector and the number of the plurality of device nodes; The third time data is determined according to the data amount of the third data, the first rate data and the first coefficient.

26. The data processing device according to claim 24, characterized in that When the data processing unit acquires the fourth time data, it is specifically used to: Acquire the data volume of the target feature vector, the number of the plurality of device nodes, second rate data, and the first coefficient; wherein the second rate data is used to indicate the data transmission rate between the first device node and device nodes other than the first device node; Determining the data volume of the fourth data according to the data volume of the target feature vector and the number of the plurality of device nodes; The fourth time data is determined according to the data amount of the fourth data, the second rate data and the first coefficient.

27. The data processing device according to any one of claims 20 to 26, characterized in that: The target feature vector is any one of the N feature vectors; wherein the N feature vectors constitute the second feature vector table, the second feature vector table is used to train the AI ​​model, the training process of the AI ​​model includes N training stages, the N feature vectors correspond one-to-one to the N training stages, each of the N feature vectors is used as training data for a corresponding training stage, and N is a positive integer.

28. The data processing device according to claim 27, characterized in that Before acquiring the plurality of feature vector tables to be stored, the data acquisition unit is further used to: Determine the start time of a set period; wherein the set period includes at least one of the training stages.

29. A data processing device, characterized in that: comprising a memory and at least one processor; The memory is used to store computer programs; The processor is configured to execute the computer program stored in the memory to implement the method according to any one of claims 1 to 14.

30. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed on a data processing device, the data processing device is caused to execute the method according to any one of claims 1 to 14.

Images