A sample feature generation system, method and apparatus

Abstract

Embodiments of the present application provide a sample feature generation system, method and device, and relate to the technical field of data processing. The system comprises a management node, a reading node and a processing node. The management node is configured to obtain target operators from existing operators according to the function of a machine learning model to be trained, and determine the execution order of each target operator; deploy each target operator to a target node selected from the processing nodes, and notify each target node of path information of its downstream nodes based on the execution order. The reading node is configured to obtain sample data without generated sample features from an external disk one by one, and send the sample data obtained each time to a first node. The first node and a second node are configured to store the received data or data processing results in a memory, and process the data in the memory using the deployed target operators to obtain sample features. The scheme provided by the embodiments of the present application can improve the generation efficiency of sample features.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a sample feature generation system, method and apparatus. Background Technology

[0002] Machine learning models have promising applications and are increasingly being used in various scenarios. For example, in business scenarios such as search, recommendation, and advertising, machine learning models are often used to make personalized predictions of user behavior. Based on the prediction results, personalized search, recommendation, and advertising services are provided to users. This can improve user experience and user stickiness to the platform, which is conducive to enhancing the platform's competitiveness.

[0003] In some cases, the input information for machine learning models is the features of the data. Furthermore, machine learning models need to be trained before use. To train a robust machine learning model, it is often necessary to collect a large amount of sample data and process this data to obtain sample features for training the model. However, when the sample data is massive and diverse, processing the sample data to obtain sample features is inefficient. Summary of the Invention

[0004] The purpose of this invention is to provide a sample feature generation system, method, and apparatus to improve the efficiency of processing raw data to generate sample features. The specific technical solution is as follows:

[0005] In a first aspect, embodiments of the present invention provide a sample feature generation system, the system comprising: a management node, a reading node, and a processing node; wherein...

[0006] The management node is used to obtain target operators from existing operators based on the function of the machine learning model to be trained, and determine the execution order of each target operator; select target nodes corresponding to each target operator from the processing nodes, deploy each target operator to the corresponding target node, and notify each target node of the path information of its downstream nodes based on the execution order.

[0007] The reading node is used to obtain sample data without generated sample features from the external disk one by one, and send the obtained sample data to the first node each time. The first node is the target node corresponding to the first target operator in the execution order.

[0008] The first node is used to store the received sample data in memory, process the sample data in memory using the deployed target operator to obtain a first result, and send the first result to its downstream nodes;

[0009] The second node is used to store the first result sent by its upstream node into memory, and to process the first result in memory using the deployed target operator to obtain the second result. If the second node has a downstream node, it sends the second result to its downstream node. If the second node does not have a downstream node, it determines the second result as the sample feature of the sample data. The second node is any target node other than the first node.

[0010] Secondly, embodiments of the present invention provide a sample feature generation method, applied to a target node in a sample feature generation system. The sample generation system includes a management node, a reading node, and a processing node. The target node is a node corresponding to each target operator selected by the management node from the processing node. The target operator is an operator obtained from existing operators based on the function of the machine learning model to be trained. The target node is deployed with corresponding target operators.

[0011] The method includes:

[0012] If the target node is the first node, it receives sample data without generated sample features obtained by the reading node from the external disk each time, stores the received sample data in memory, processes the sample data in memory using the deployed target operator to obtain a first result, and sends the first result to its downstream node so that the downstream node generates sample features of the sample data based on the first result. Here, the first node is the target node corresponding to the first target operator in the execution order, and the downstream node is the node determined according to the execution order.

[0013] If the target node is a second node, the first result sent by its upstream node is stored in memory, and the first result in memory is processed by the deployed target operator to obtain the second result. If the second node has a downstream node, the second result is sent to its downstream node. If the second node does not have a downstream node, the second result is determined as the sample feature of the sample data. The second node is any target node other than the first node.

[0014] Thirdly, embodiments of the present invention provide a sample feature generation method, which applies a management node in a sample feature generation system, the sample feature generation system including: a management node, a reading node, and a processing node;

[0015] The method includes:

[0016] Based on the function of the machine learning model to be trained, the target operators are obtained from the existing operators, and the execution order of each target operator is determined.

[0017] Select the target node corresponding to each target operator from the processing node, and deploy each target operator to the corresponding target node;

[0018] Based on the execution order, the path information of the downstream nodes is notified to each target node, so that each target node can use its deployed target operators to process the sample data sent by the reading node in turn to generate sample features according to the execution order.

[0019] Fourthly, embodiments of the present invention provide a sample feature generation device applied to a target node in a sample feature generation system. The sample generation system includes a management node, a reading node, and a processing node. The target node is a node corresponding to each target operator selected by the management node from the processing node. The target operator is an operator obtained from existing operators based on the function of the machine learning model to be trained. The target node is deployed with corresponding target operators.

[0020] The device includes:

[0021] The first feature generation module is configured to, if the target node is the first node, receive sample data without generated sample features obtained by the reading node from the external disk each time, store the received sample data in memory, process the sample data in memory using the deployed target operator to obtain a first result, and send the first result to its downstream node so that the downstream node generates sample features of the sample data based on the first result, wherein the first node is the target node corresponding to the first target operator in the execution order, and the downstream node is the node determined according to the execution order;

[0022] The second feature generation module is used to store the first result sent by its upstream node into memory if the target node is a second node, and process the first result in memory using the deployed target operator to obtain the second result. If the second node has a downstream node, the module sends the second result to its downstream node. If the second node does not have a downstream node, the module determines the second result as the sample feature of the sample data. The second node is any target node other than the first node.

[0023] Fifthly, embodiments of the present invention provide a sample feature generation device that utilizes a management node in a sample feature generation system, the sample feature generation system comprising: a management node, a reading node, and a processing node;

[0024] The device includes:

[0025] The operator acquisition module is used to obtain target operators from existing operators based on the functionality of the machine learning model to be trained, and to determine the execution order of each target operator.

[0026] The node determination module selects the target node corresponding to each target operator from the processing nodes and deploys each target operator to the corresponding target node;

[0027] The information notification module is used to notify each target node of the path information of its downstream nodes based on the execution order, so that each target node can use its deployed target operators to process the sample data sent by the reading node in turn according to the execution order to generate sample features.

[0028] In a sixth aspect, embodiments of the present invention provide an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus;

[0029] Memory, used to store computer programs;

[0030] When a processor executes a program stored in memory, it implements the steps of the method described in the second or third aspect above.

[0031] In a seventh aspect, embodiments of the present invention provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the method described in the second or third aspect above.

[0032] Eighthly, embodiments of the present invention provide a computer program product containing instructions that, when run on a computer, cause the computer to perform the steps of the method described in any of the second or third aspects above.

[0033] As can be seen from the above, in the sample feature generation scheme provided by the embodiments of the present invention, after the management node obtains each target operator and deploys the target operators to the target node, the reading node can obtain sample data without generated sample features from the external disk one by one, and send the sample data obtained each time to the first node in the target node that processes the sample data. The first node can store the received data in memory, and use the deployed target operator to process the sample data in memory to obtain a first result and send the first result to its downstream node. In this way, the second node in the target node other than the first node can store the first result sent by its upstream node in memory, so that it can send the second result obtained by processing the first result to its downstream node when it has a downstream node, and determine the second result as the sample feature of the sample data when it does not have a downstream node.

[0034] As can be seen, when generating sample features, only the read node needs to read data from the external disk sequentially and send the read data to the first node. For the first node, it can directly store the received data into memory and process it to obtain the first result. For the second node, it can directly store the received first result into memory and process it to obtain the second result. That is, both the first and second nodes can process data directly in memory without reading data from the external disk, saving the time consumed when reading data from the external disk. In practical scenarios, there are often many first and second nodes used to process sample data. When generating sample features using this solution, only one read node needs to read data from the external disk, instead of a large number of first and second nodes reading data from the external disk. This greatly reduces the frequency of nodes reading data from the external disk when generating sample features, reduces the time consumed when nodes read data from the external disk, and improves the efficiency of sample feature generation. Attached Figure Description

[0035] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below.

[0036] Figure 1 This is a schematic diagram of the structure of a sample feature generation system provided in an embodiment of the present invention;

[0037] Figure 2 Signaling interaction diagram of the first sample feature generation method provided in the embodiments of the present invention;

[0038] Figure 3a A schematic diagram of the first sample feature generation process provided in an embodiment of the present invention;

[0039] Figure 3b This is a schematic diagram of the second sample feature generation process provided in an embodiment of the present invention;

[0040] Figure 4 A flowchart illustrating the second sample feature generation method provided in this embodiment of the invention;

[0041] Figure 5 A flowchart illustrating the third sample feature generation method provided in this embodiment of the invention;

[0042] Figure 6 A flowchart illustrating the fourth sample feature generation method provided in this embodiment of the invention;

[0043] Figure 7 This is a schematic diagram of the structure of the first sample feature generation device provided in an embodiment of the present invention;

[0044] Figure 8 This is a schematic diagram of the structure of the second sample feature generation device provided in an embodiment of the present invention;

[0045] Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0046] The technical solutions of the present invention will now be described with reference to the accompanying drawings in the embodiments of the present invention.

[0047] To improve the efficiency of sample feature generation, embodiments of the present invention provide a sample feature generation system, method, and apparatus, which are described in detail below.

[0048] First, this invention provides a sample feature generation system, which includes: a management node, a reading node, and a processing node;

[0049] The aforementioned management node is used to obtain target operators from existing operators based on the function of the machine learning model to be trained, and determine the execution order of each target operator; select the target node corresponding to each target operator from the processing node, deploy each target operator to the corresponding target node, and notify each target node of the path information of its downstream nodes based on the aforementioned execution order.

[0050] The aforementioned read nodes are used to sequentially obtain sample data without generated sample features from the external disk and send the obtained sample data to the first node each time; wherein, the aforementioned first node is the target node corresponding to the first target operator in the aforementioned execution order;

[0051] The aforementioned first node is used to store the received sample data in memory, process the sample data in memory using the deployed target operator to obtain a first result, and send the first result to its downstream nodes;

[0052] The second node is used to store the first result sent by its upstream node into memory, and to process the first result in memory using the deployed target operator to obtain the second result. If the second node has a downstream node, it sends the second result to its downstream node. If the second node does not have a downstream node, it determines the second result as the sample feature of the sample data. The second node is a target node other than the first node.

[0053] As can be seen from the above, in the sample feature generation scheme provided by the embodiments of the present invention, after the management node obtains each target operator and deploys the target operators to the target node, the reading node can obtain sample data without generated sample features from the external disk one by one, and send the sample data obtained each time to the first node in the target node that processes the sample data. The first node can store the received data in memory, and use the deployed target operator to process the sample data in memory to obtain a first result and send the first result to its downstream node. In this way, the second node in the target node other than the first node can store the first result sent by its upstream node in memory, so that it can send the second result obtained by processing the first result to its downstream node when it has a downstream node, and determine the second result as the sample feature of the sample data when it does not have a downstream node.

[0054] See Figure 1 , Figure 1 This is a schematic diagram of the structure of a sample feature generation system provided in an embodiment of this application. The sample feature generation system includes: a management node 101, a processing node 102, and a reading node 103.

[0055] All of the above nodes can be any electronic device with data processing, communication, or other functions.

[0056] In one scenario, the aforementioned nodes could be servers deployed on a feature production platform.

[0057] To more intuitively illustrate how the above-mentioned sample feature generation system generates sample features, the following uses a signaling diagram to provide a detailed explanation of the sample feature generation scheme provided in this application embodiment.

[0058] See Figure 2 This is a signaling interaction diagram of the first sample feature generation method provided in the embodiments of the present invention. The method includes the following steps S201-S208.

[0059] Step S201: The management node obtains the target operator from the existing operators based on the function of the machine learning model to be trained, and determines the execution order of each target operator.

[0060] The embodiments of the present invention do not limit the specific type, training method or model function of the machine learning model to be trained.

[0061] For example, from the perspective of model type, machine learning models can be decision tree models, shallow neural network models, deep neural network models, etc.; from the perspective of training method, machine learning models can be supervised models, unsupervised models, etc.; from the perspective of model function, machine learning models can be image recognition models, speech recognition models, object detection models, etc.

[0062] The existing operators are pre-packaged operators used to process sample data.

[0063] Specifically, the existing operators mentioned above may include at least one of the following operators:

[0064] Data read operator, data write operator, data filter operator, data deduplication operator, data type conversion operator, data augmentation operator, and data merging operator.

[0065] It is evident that existing operators can include a wide variety of operators, enabling various data processing methods and meeting the processing needs of diverse sample data.

[0066] It should be noted that the above operators are only some examples. The embodiments of the present invention do not limit the specific functions of existing operators. Existing operators may include operators used to perform arbitrary processing operations on sample data.

[0067] The target operator is an operator selected from existing operators, used to process sample data to obtain sample features for training machine learning models.

[0068] Specifically, the management node can determine candidate operators from existing operators based on the functionality of the machine learning model to be trained, and then determine the target operators and the execution order of each target operator based on the candidate operators.

[0069] For example, it is possible to obtain the target operators selected by the user based on the candidate operators, as well as the execution order of each target operator. The user mentioned above is the user of the machine learning model to be trained.

[0070] First, the method for determining candidate operators will be explained.

[0071] The management node can pre-establish the correspondence between the model functions of the machine learning model and the candidate operators. In this way, the corresponding candidate operators can be selected according to the functions of the machine learning model to be trained based on the above correspondence.

[0072] For example, when the model function is image classification, the corresponding candidate operators are image processing-related candidate operators, such as image denoising operators, image grayscale adjustment operators, image segmentation operators, and image feature extraction operators; when the model function is text recognition, the corresponding candidate operators are text processing-related candidate operators, such as text segmentation operators, character recognition operators, and semantic prediction operators.

[0073] Since the target operator is used to obtain sample features, and model features are often closely related to model function, this method identifies candidate operators related to the model function of the machine learning model from a large number of existing operators. This avoids interference from operators unrelated to the model function when selecting the target operator, which is beneficial for selecting the target operator efficiently.

[0074] After obtaining the candidate operators, the management node can obtain the target operators determined by the user based on the candidate operators and the execution order of each target operator in the following two cases.

[0075] In one scenario, the management node can directly obtain the identifiers of the target operators provided by the user, as well as the execution order information corresponding to each identifier.

[0076] In this scenario, if the candidate operators are insufficient to meet the user's needs, the management node can provide the user with more existing operators based on the user's request. In this way, the management node can identify the operator corresponding to the aforementioned identifier from the existing operators as the target operator, and determine the execution order of each target operator based on the execution order information corresponding to each identifier.

[0077] In another scenario, the management node can obtain relational data and determine the target operators and their execution order based on this data. Detailed implementation methods are described in subsequent embodiments and will not be elaborated upon here.

[0078] Step S202: The management node selects the target node corresponding to each target operator from the processing nodes.

[0079] In this step, the management node can select the target node corresponding to each target operator from the processing nodes in several ways. Examples are given below:

[0080] In the first approach, the management node can select the target node corresponding to each target operator based on the data processing status of each processing node.

[0081] For example, for each target operator, the management node can randomly select a processing node from the processing nodes whose current data processing status is idle as the target node corresponding to that target operator.

[0082] In the second approach, the management node can select the target node corresponding to each target operator based on the expected amount of computing resources required by each target operator and the current available computing resources of each processing node.

[0083] In some cases, the expected computational resources required by different target operators can be different. For example, a data deduplication operator needs to traverse all the data, while a data merging operator may only need to concatenate the data to be merged. The expected computational resources required by these two operators can be different.

[0084] Therefore, the management node can determine the expected amount of computing resources for each target operator, and then select a processing node whose current available computing resources are not less than the expected computing resources of the target operator as its corresponding target processing node.

[0085] For example, for each target operator, the management node can randomly select a node from the processing nodes whose current available computing resources are greater than the expected computing resources of the target operator, and use it as the target node for that target operator.

[0086] For example, the management node can sort the target operators from highest to lowest according to the expected amount of computing resources for each target operator, and then sort the processing nodes from highest to lowest according to the amount of currently available computing resources. Then, based on the sorting results of the above two sorting methods, the processing node with the same sorting position as each target operator is determined as its corresponding target node.

[0087] Step S203: The management node deploys each target operator to the corresponding target node and notifies each target node of the path information of its downstream nodes based on the execution order.

[0088] The management node can deploy each target operator to the corresponding target node in the following ways.

[0089] In one implementation, the management node can remotely control the target node and deploy each target operator to the target node.

[0090] In another implementation, the management node can send its corresponding target operator to the target node, so that each target node will deploy the received target operator locally.

[0091] After the management node deploys each target operator to the corresponding target node, it can send the path information of its downstream nodes to each target node, so that each target node can access its downstream nodes based on the path information.

[0092] Specifically, the aforementioned path information can be information such as the MAC address and IP address of its downstream nodes that can uniquely identify the node's access path.

[0093] Step S204: The reading node successively obtains sample data without generated sample features from the external disk and sends the obtained sample data to the first node each time.

[0094] The first node is the target node corresponding to the first target operator in the execution order.

[0095] Specifically, after the management node deploys each target operator to its corresponding target node, it can determine the target node corresponding to the first target operator in the execution order as the first node and notify the reading node of the information of the first node.

[0096] First, we will explain when the read node reads sample data from the external disk.

[0097] In one implementation, the reading node can read sample data from the external disk sequentially at preset reading intervals when each reading opportunity arrives.

[0098] In another implementation, the reading node can read sample data from the external disk each time it receives a data acquisition request from the first node.

[0099] The following explains how to determine sample data for which no sample features have been generated when the read node reads data sequentially.

[0100] Specifically, the reading node can mark sample data that has been sent on the external disk, and then identify unmarked sample data as sample data that has not generated sample features when reading data subsequently.

[0101] Next, we will explain the amount of data that the read node reads from the external disk each time.

[0102] In one implementation, the amount of data read by the reading node from the external disk each time can be set by the staff.

[0103] In another implementation, the reading node can determine the amount of data to be read from the external disk each time based on the current available memory storage of the first node.

[0104] For example, when the first node sends a data retrieval request to the reading node, it can include its current available memory storage. The reading node can then determine the amount of data to be read each time as 50%, 80%, or other values ​​of the available memory storage.

[0105] As can be seen from the above, the reading node reads sample data from the external disk one by one. The amount of data to be read each time can be determined based on the current available memory storage of the first node. This allows the first node to process the data directly in memory by sending the sample data read each time.

[0106] Step S205: The first node stores the received sample data in memory and processes the sample data in memory using the deployed target operator to obtain the first result.

[0107] As can be seen from the foregoing embodiments, the reading node sends sample data to the first node sequentially, and the amount of data sent each time can be determined based on the current available memory storage capacity of the first node.

[0108] Therefore, the first node can directly store the received sample data in memory, and then use the deployed target operator to process the sample data in memory to obtain the first result.

[0109] Step S206: The first node sends the first result to its downstream nodes.

[0110] As can be seen from the foregoing embodiments, the management node notifies each target node of its downstream nodes, so that after the first node obtains the first result, it can send the first result to its downstream nodes.

[0111] Step S207: The second node stores the first result sent by its upstream node into memory, and processes the first result in memory using the deployed target operator to obtain the second result.

[0112] The second node is the target node other than the first node.

[0113] The upstream node of the second node is the target node that sends the first result to the second node.

[0114] Step S208: The second node determines whether it has a downstream node. If yes, it sends the second result to its downstream node. If no, it determines the second result as the sample feature of the sample data.

[0115] If a second node determines that it has downstream nodes, it indicates that there are still target nodes processing the sample data, and the sample data processing flow is not yet complete. The second result obtained by the second node after processing the received first result is not the final result of the sample data processing. Therefore, the second node can send its obtained second result to its downstream nodes, so that the downstream nodes can continue processing based on the second result to obtain the sample features.

[0116] If the second node determines that it has no downstream nodes, it indicates that this second node is the target node corresponding to the last target operator in the execution sequence. Since there are no more target nodes processing the sample data, the sample data processing flow ends. The second result obtained by this second node after processing the received first result is the final processed result of the sample data. Therefore, this second node can determine the second result as the final processed result of the sample data, that is, determine the second result as the sample feature.

[0117] As can be seen from the above, in the sample feature generation scheme provided by the embodiments of the present invention, after the management node obtains each target operator and deploys the target operators to the target node, the reading node can obtain sample data without generated sample features from the external disk one by one, and send the sample data obtained each time to the first node in the target node that processes the sample data. The first node can store the received data in memory, and use the deployed target operator to process the sample data in memory to obtain a first result and send the first result to its downstream node. In this way, the second node in the target node other than the first node can store the first result sent by its upstream node in memory, so that it can send the second result obtained by processing the first result to its downstream node when it has a downstream node, and determine the second result as the sample feature of the sample data when it does not have a downstream node.

[0118] As can be seen, when generating sample features, only the read node needs to read data from the external disk sequentially and send the read data to the first node. For the first node, it can directly store the received data into memory and process it to obtain the first result. For the second node, it can directly store the received first result into memory and process it to obtain the second result. That is, both the first and second nodes can process data directly in memory without reading data from the external disk, saving the time consumed when reading data from the external disk. In practical scenarios, there are often many first and second nodes used to process sample data. When generating sample features using this solution, only one read node needs to read data from the external disk, instead of a large number of first and second nodes reading data from the external disk. This greatly reduces the frequency of nodes reading data from the external disk when generating sample features, reduces the time consumed when nodes read data from the external disk, and improves the efficiency of sample feature generation.

[0119] Furthermore, when generating sample features using the scheme provided in this embodiment of the invention, the target operator deployed on the target node is a target operator selected from existing operators, rather than user-provided custom code, thus eliminating the need for users to write code. When the processing node uses the deployed target operator to process data, it actually calls the code encapsulated in the target operator to process the data. Therefore, during subsequent maintenance or upgrades of the feature production platform, only the existing operators need to be updated in a timely manner, allowing each processing node in the feature production platform to continue processing the data by calling the updated code encapsulated in the target operator. This eliminates the need to adjust the user-defined code deployed on each processing node individually, and also eliminates the need for users to re-upload their code, reducing the difficulty and cost of upgrading and maintaining the feature production platform.

[0120] The following describes another scenario mentioned in step S201 above, where the target operator is determined after obtaining the candidate operator, and the execution order among the target operators is determined.

[0121] The management node can obtain relational data, identify the operators corresponding to relational objects as target operators, and determine the execution order of each target operator based on the relationships between relational objects.

[0122] Among them, the operators corresponding to the relation objects recorded in the relation data are: the operators in the candidate operators, and the relationship between relation objects represents the execution order between the operators corresponding to the relation objects;

[0123] In this way, the management node can obtain the target operator and the execution order of the target operator through the above relational data.

[0124] Specifically, the management node may include a user interface that allows users to combine target operators by dragging and dropping icons representing target operators, setting directed line segments between icons, etc., and in response to user clicks, generates relational data representing the user-combined operators and the execution order between operators.

[0125] The following example illustrates the format of the aforementioned relational data.

[0126] In one embodiment of the present invention, the relational data may define the operators corresponding to the relational objects, the hierarchical traversal relationship of the operators corresponding to the relational objects, and the specific execution logic of the operators corresponding to the relational objects.

[0127] This allows us to parse the computational dependencies between operators based on the dependencies between relational objects in the relational data, thereby obtaining the execution order of the operators.

[0128] In this way, the management platform can efficiently obtain the target operators and their execution order based on the relational data described above.

[0129] In one embodiment of the present invention, the management node in the sample feature generation system is further configured to respond to an operator addition request, obtain a custom operator for processing sample data, and add the custom operator to the existing operators.

[0130] In situations where existing operators are insufficient to meet user needs or fail to perform the data transformation operations expected by the user, the management node can add user-provided custom operators to the existing operators for the user to choose from, thus enabling the user to meet their needs. Furthermore, this allows for continuous updates and expansion of existing operators, which is beneficial for meeting more data processing requirements.

[0131] In one scenario, after responding to an operator addition request and obtaining a custom operator for processing sample data, the management node can determine whether the format of the custom operator is a preset operator format. If so, it adds the custom operator to the existing operators. The preset operator format is the operator format of the existing operators.

[0132] The following example illustrates the preset operator format.

[0133] In one embodiment of the present invention, the preset operator may define the operator type, the specific calculation logic of the operator, and the execution result returned after the operator is executed.

[0134] Next, we will provide an example of a custom operator written in the above format.

[0135] For example, if the custom operator is a data filtering operator, you can first define the operator type as a filtering operator, then encapsulate the specific calculation logic of the filtering operator, and define how to return the execution result after executing the operator. The specific calculation logic of the filtering operator could be something like swapping the rows and columns of the input data matrix.

[0136] This ensures that the format of the added custom operators is consistent with that of the existing operators, making it easier to update the operators when the feature generation platform is updated or upgraded in the future.

[0137] In one embodiment of the present invention, the sample feature generation system further includes a storage node mounted on an external disk. In this case, the second node is also used to send the determined sample features to the storage node in the absence of a downstream node. The storage node is used to store the received sample features to the mounted external disk.

[0138] After receiving the sample features, the storage node can store the sample features on an external disk located at a pre-configured disk path. This allows the model to be trained by directly reading the sample features from the external disk.

[0139] This way, a dedicated storage node stores the sample features to an external disk, allowing the target node that processes the sample data to focus on generating sample features without spending extra time storing the generated features, which helps improve the efficiency of sample feature generation.

[0140] The following is a detailed explanation of the differences between the sample feature generation scheme provided in the embodiments of the present invention and the sample feature generation scheme in the prior art.

[0141] To illustrate the differences more clearly, the following will be combined with... Figure 3a and Figure 3b Perform the analysis.

[0142] See Figure 3a , Figure 3a This is a comparative schematic diagram of the first sample feature generation process provided in an embodiment of the present invention. Figure 3a The process shown is the sample feature generation process in the prior art.

[0143] Figure 3a A dashed box represents a task, a solid box represents a node, an arrow pointing to the left represents a disk read operation, a cube represents a logical computation, an arrow pointing to the right represents a disk write operation, or disk storage operation, and a thick dashed line pointing downwards represents control dependencies, which can also be understood as the execution order of nodes.

[0144] Depend on Figure 3a As can be seen, the disk mounted on node n stores the data sent by its upstream nodes. Node n first reads the data from the disk, then performs logical calculations, writes the calculated data to the disk, and then sends the written data to subsequent nodes, such as node n+1. Node n+1 receives the data and stores it on the disk, then reads the data from the disk and performs logical calculations, writes the calculated data to the disk, and then sends the data written to the disk to subsequent nodes, such as node n+2, and so on, until the data processing flow is completed. The last node obtains the sample features and stores them on the disk.

[0145] As can be seen, due to the limited memory of each node, each node stores the received data to an external disk, and then reads and processes the data from the external disk. In other words, there is no direct data flow between the memory locations of the nodes. Each node needs to perform disk read and disk write operations; this can be understood as each node corresponding to a relatively independent task.

[0146] This results in a high frequency of disk read / write operations during the entire sample feature generation process. Since there are many nodes in the actual scenario and the nodes read and write data from the disk at a low rate, a lot of disk read / write time is required, resulting in low efficiency in generating sample features.

[0147] The sample feature generation process provided in the embodiments of the present invention will be further explained.

[0148] See Figure 3b , Figure 3b This is a comparative schematic diagram of the second sample feature generation process provided in the embodiments of the present invention. Figure 3b The process shown is the sample feature generation process provided in the embodiment of the present invention.

[0149] Figure 3b A thick solid arrow pointing downwards from the middle indicates data flow; the meanings of other rectangles and arrows are the same. Figure 3aThe same applies, so I won't go into details here.

[0150] Depend on Figure 3b As can be seen, in this scheme, the reading node is responsible for reading data from the disk sequentially and sending it to the first node. The first node directly stores the received data in memory and processes the data in memory using the deployed target operator, then sends the processed data to the second node 1. The second node 1 directly stores the received data in memory and processes the data in memory using the deployed target operator, then sends the processed data to its downstream node, the second node 2, and so on, until the second node N, which has no downstream node, processes the data and sends the obtained sample features to the storage node. The storage node stores the received sample features on the disk.

[0151] As can be seen, in the entire sample feature generation process, the reading node is responsible for reading data from the external disk in multiple batches and sending the data to the first node, which can directly store the data in its memory; the first and second nodes are responsible for processing the data directly in memory and sending the processing results to downstream nodes or storage nodes; the storage node is responsible for storing the generated sample features. This can also be understood as all types of nodes collaborating as a whole to handle the same sample feature generation task.

[0152] As can be seen from the comparison, compared with the prior art, the sample feature generation scheme provided in this application embodiment only requires the reading node and the storage node to perform disk read or write operations. For a large number of first nodes and second nodes, the data flows in memory and does not need to perform disk read and write operations, which reduces the disk read and write frequency of the sample feature generation process, thereby reducing the time required for nodes to perform disk read and write operations and improving the efficiency of the entire sample feature generation process.

[0153] Therefore, compared with the prior art, under the same conditions, the sample feature generation scheme provided in this application embodiment can generate sample features faster and provide them for subsequent machine learning model training.

[0154] Corresponding to the above-mentioned sample feature generation system, this embodiment of the invention also provides sample feature generation methods for target nodes and management nodes respectively applied to the sample feature generation system, which will be described below.

[0155] First, we introduce the sample feature generation method applied to the target node in the sample generation system.

[0156] As can be seen from the foregoing embodiments, the target node can be either the first node or the second node. The following will illustrate this further. Figure 4 , Figure 5 They will be introduced separately.

[0157] See Figure 4This is a flowchart illustrating the second sample feature generation method provided in this embodiment of the invention. It is applied to the first node, and the method includes the following steps S401-S403.

[0158] Step S401: Receive sample data of ungenerated sample features obtained by the read node from the external disk each time.

[0159] Step S402: Store the received sample data in memory, and use the deployed target operator to process the sample data in memory to obtain the first result.

[0160] Step S403: Send the first result to its downstream nodes so that the downstream nodes can generate sample features of the sample data based on the first result.

[0161] The first node can have multiple downstream nodes. The method by which the downstream nodes generate sample features based on the first result will be detailed later. Figure 5 The illustrated embodiment.

[0162] As can be seen from the above, the first node can receive sample data without generated sample features obtained by the reading node from the external disk each time, store the received data in memory, process the sample data in memory using the deployed target operator to obtain the first result, and send the first result to its downstream node so that the downstream node can generate sample features of the sample data based on the first result.

[0163] As can be seen, when generating sample features, for the first node, it can directly store the received data into memory and process it to obtain the first result without having to read data from an external disk, thus saving the time consumed when reading data from an external disk and improving the efficiency of sample feature generation.

[0164] See Figure 5 This is a flowchart illustrating the third sample feature generation method provided in this embodiment of the invention, applied to the second node. The method includes the following steps S501-S504.

[0165] Step S501: Store the first result sent by its upstream node in memory, and use the deployed target operator to process the first result in memory to obtain the second result.

[0166] Step S502: Determine if there is a downstream node. If yes, proceed to step S503; otherwise, proceed to step S504.

[0167] Step S503: Send the second result to its downstream nodes.

[0168] Downstream nodes are determined based on the execution order.

[0169] Specifically, as can be seen from the foregoing embodiments, the management node can notify the downstream nodes of each target node according to the execution order of the target operators.

[0170] If the second node determines that it has downstream nodes, it means that there are still target nodes to process the sample data, and the sample data processing process has not ended. The second result obtained by the second node after processing the first result received is not the final processing result of the sample data.

[0171] Therefore, the second node can execute step S503, sending the second result it obtained to its downstream node, so that its downstream node can continue processing based on the second result to obtain sample features.

[0172] Step S504: Determine the second result as the sample feature of the sample data.

[0173] If the second node determines that it has no downstream nodes, it means that the second node is the target node corresponding to the last target operator in the execution order. There are no more target nodes to process the sample data, the sample data processing flow ends, and the second result obtained by the second node after processing the first result received is the final processing result of the sample data.

[0174] Therefore, the second node can execute step S504 to determine the second result as the final processing result of the sample data, that is, to determine the second result as the sample feature.

[0175] As can be seen from the above, the second node can store the first result sent by its upstream node in memory, send the second result obtained by processing the first result to its downstream node when there is a downstream node, and determine the second result as the sample feature of the sample data when there is no downstream node.

[0176] As can be seen, when generating sample features, the second node can directly store the received first result in memory and process it to obtain the second result, without needing to read data from an external disk, thus saving the time consumed by reading data from an external disk. In real-world scenarios, there are often multiple second nodes used to process sample data. When using this solution to generate sample features, there is no need for a large number of second nodes to read data from external disks. This reduces the frequency of nodes reading data from external disks during the entire sample feature generation process, reduces the time consumed by nodes reading data from external disks, and improves the efficiency of sample feature generation.

[0177] In one embodiment of the present invention, the sample generation system further includes a storage node mounted with an external disk. In this case, the above method further includes:

[0178] In the absence of downstream nodes, the second node can send the determined sample features to the storage node, so that the storage node can store the received sample features to the mounted external disk.

[0179] This way, a dedicated storage node stores the sample features to an external disk, allowing the target node that processes the sample data to focus on generating sample features without spending extra time storing the generated features, which helps improve the efficiency of sample feature generation.

[0180] Next, we will introduce the sample feature generation method applied to management nodes.

[0181] See Figure 6 , Figure 6 This is a flowchart illustrating the fourth sample feature generation method provided in this embodiment of the invention, applied to a management node. The method includes the following steps S601-S603.

[0182] Step S601: Based on the function of the machine learning model to be trained, obtain the target operator from the existing operators and determine the execution order of each target operator.

[0183] This step has already been explained in step S201 above, and will not be repeated here.

[0184] Step S602: Select the target node corresponding to each target operator from the processing nodes, and deploy each target operator to the corresponding target node.

[0185] This step has already been explained in step S202 above, and will not be repeated here.

[0186] Step S603: Based on the execution order, notify each target node of the path information of its downstream nodes, so that each target node can use its deployed target operators to process the sample data sent by the reading node in turn to generate sample features according to the execution order.

[0187] For details of this step, please refer to the aforementioned implementation examples applied to the first and second nodes, which will not be repeated here.

[0188] As can be seen from the above, the management node can obtain target operators from existing operators and determine the execution order of each target operator based on the function of the machine learning model to be trained. This allows the management node to deploy each target operator to the target node selected from the processing node and notify each target node of the path information of its downstream nodes based on the execution order. This enables each target node to use its deployed target operators to process the sample data sent by the reading node in turn according to the execution order to generate sample features.

[0189] As can be seen, when generating sample features, the data processed by the target nodes is sent sequentially by the reading nodes, rather than being read from external disks, thus saving the time consumed when reading data from external disks. In real-world scenarios, there are often many target nodes used to process sample data. When generating sample features using this solution, it is not necessary for a large number of target nodes to read data from external disks. This significantly reduces the frequency of nodes reading data from external disks during the entire sample feature generation process, reduces the time consumed by nodes reading data from external disks, and improves the efficiency of sample feature generation.

[0190] In one embodiment of the present invention, step S601 can also be implemented by the following steps:

[0191] Based on the function of the machine learning model to be trained, candidate operators are determined from the existing operators, the operators corresponding to the relation objects recorded in the relation data are determined as target operators, and the execution order of each target operator is determined according to the relationship between the relation objects.

[0192] For detailed implementation methods, please refer to the foregoing. Figure 1 The following embodiments will not be described in detail here.

[0193] Since the target operator is used to obtain sample features, and model features are often closely related to model function, this method identifies candidate operators related to the model function of the machine learning model from a large number of existing operators. This avoids interference from operators unrelated to the model function when selecting the target operator, which is beneficial for selecting the target operator efficiently.

[0194] In one embodiment of the present invention, the method further includes:

[0195] In response to an operator addition request, a custom operator for processing the sample data is obtained, and the custom operator is added to the existing operators.

[0196] In this way, when the existing operators are insufficient to meet user needs, the management node can add user-provided custom operators to the existing operators for users to choose from, so that users can meet their own needs; and this can continuously update and expand the existing operators, which is conducive to meeting more data processing needs.

[0197] Corresponding to the above-mentioned sample feature generation system, this embodiment of the invention also provides sample feature generation devices for target nodes and management nodes, which will be described below.

[0198] See Figure 7This is a schematic diagram of the structure of a first sample feature generation device provided in an embodiment of the present invention, applied to a target node in a sample feature generation system. The device includes the following modules 701-702:

[0199] The first feature generation module 701 is configured to, if the target node is the first node, receive sample data without generated sample features obtained by the reading node from the external disk each time, store the received sample data in memory, process the sample data in memory using the deployed target operator to obtain a first result, and send the first result to its downstream node so that the downstream node generates sample features of the sample data according to the first result, wherein the first node is the target node corresponding to the first target operator in the execution order, and the downstream node is the node determined according to the execution order;

[0200] The second feature generation module 702 is used to store the first result sent by its upstream node into memory if the target node is a second node, and process the first result in memory using the deployed target operator to obtain a second result. If the second node has a downstream node, the second result is sent to its downstream node. If the second node does not have a downstream node, the second result is determined as a sample feature of the sample data. The second node is any target node other than the first node.

[0201] As can be seen from the above, the first node can receive sample data without generated sample features obtained from the external disk by the reading node each time, store the received data in memory, process the sample data in memory using the deployed target operator to obtain the first result, and send the first result to its downstream node; the second node can store the first result sent by its upstream node in memory, send the second result obtained by processing the first result to its downstream node when it has a downstream node, and determine the second result as the sample feature of the sample data when it does not have a downstream node.

[0202] As can be seen, when generating sample features, the first node can directly store the received data into memory and process it to obtain the first result; similarly, the second node can directly store the received first result into memory and process it to obtain the second result. In other words, both the first and second nodes can process data directly in memory without needing to read data from an external disk, thus saving the time consumed when reading data from an external disk. In real-world scenarios, there are often many first and second nodes used to process sample data. When using this solution to generate sample features, it eliminates the need for a large number of first and second nodes to read data from external disks. This reduces the frequency of nodes reading data from external disks during the entire sample feature generation process, decreases the time consumed when nodes read data from external disks, and improves the efficiency of sample feature generation.

[0203] In one embodiment of the present invention, the sample generation system further includes: a storage node mounted with an external disk;

[0204] The device further includes:

[0205] The feature sending module is used to send the determined sample features to the storage node when the target node is a second node and there is no downstream node, so that the storage node stores the received sample features to the mounted external disk.

[0206] This way, a dedicated storage node stores the sample features to an external disk, allowing the target node that processes the sample data to focus on feature generation without spending extra time storing the generated sample features, which helps improve the efficiency of sample feature generation.

[0207] See Figure 8 This is a schematic diagram of the structure of a second sample feature generation device provided in an embodiment of the present invention. The device utilizes a management node in the sample feature generation system and includes the following modules 801-803:

[0208] The operator acquisition module 801 is used to obtain target operators from existing operators based on the function of the machine learning model to be trained, and to determine the execution order of each target operator.

[0209] The node determination module 802 selects the target nodes corresponding to each target operator from the processing nodes and deploys each target operator to the corresponding target node;

[0210] The information notification module 803 is used to notify each target node of the path information of its downstream nodes based on the execution order, so that each target node can use its deployed target operators to process the sample data sent by the reading node in turn according to the execution order to generate sample features.

[0211] As can be seen from the above, the management node can obtain target operators from existing operators and determine the execution order of each target operator based on the function of the machine learning model to be trained. This allows the management node to deploy each target operator to the target node selected from the processing node and notify each target node of the path information of its downstream nodes based on the execution order. This enables each target node to use its deployed target operators to process the sample data sent by the reading node in turn according to the execution order to generate sample features.

[0212] As can be seen, when generating sample features, the data processed by the target nodes is sent sequentially by the reading nodes, rather than being read from external disks, thus saving the time consumed when reading data from external disks. In real-world scenarios, there are often many target nodes used to process sample data. When generating sample features using this solution, it is not necessary for a large number of target nodes to read data from external disks. This significantly reduces the frequency of nodes reading data from external disks during the entire sample feature generation process, reduces the time consumed by nodes reading data from external disks, and improves the efficiency of sample feature generation.

[0213] In one embodiment of the present invention, the operator acquisition module 801 is specifically used to determine candidate operators from existing operators based on the function of the machine learning model to be trained; obtain relation data, wherein the operators corresponding to relation objects recorded in the relation data are: the operators in the candidate operators, and the relationship between the relation objects represents the execution order between the operators corresponding to the relation objects; determine the operators corresponding to the relation objects as target operators, and determine the execution order between each target operator based on the relationship between the relation objects.

[0214] Since the target operator is used to obtain sample features, and model features are often closely related to model function, this method identifies candidate operators related to the model function of the machine learning model from a large number of existing operators. This avoids interference from operators unrelated to the model function when selecting the target operator, which is beneficial for selecting the target operator efficiently.

[0215] In one embodiment of the present invention, the apparatus further includes:

[0216] The operator adding module is used to respond to an operator adding request, obtain a custom operator for processing sample data, and add the custom operator to the existing operators.

[0217] In this way, when the existing operators are insufficient to meet user needs, the management node can add user-provided custom operators to the existing operators for users to choose from, so that users can meet their own needs; and this can continuously update and expand the existing operators, which is conducive to meeting more data processing needs.

[0218] This invention also provides an electronic device, such as... Figure 9 As shown, it includes a processor 901, a communication interface 902, a memory 903, and a communication bus 904, wherein the processor 901, the communication interface 902, and the memory 903 communicate with each other through the communication bus 904.

[0219] Memory 903 is used to store computer programs;

[0220] When the processor 901 executes the program stored in the memory 903, it implements the above-described sample feature generation method applied to the target node or the management node.

[0221] In one embodiment of the present invention, the memory 903 is further used to store sample data.

[0222] The communication bus mentioned above can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0223] The communication interface is used for communication between the aforementioned terminal and other devices.

[0224] The memory may include random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0225] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0226] In another embodiment of the present invention, a computer-readable storage medium is also provided, wherein a computer program is stored therein, and when the computer program is executed by a processor, it implements the above-described sample feature generation method applied to a target node or a management node.

[0227] In another embodiment of the present invention, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to execute the above-described sample feature generation method applied to a target node or a management node.

[0228] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid state disk (SSD)).

[0229] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0230] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments of methods, apparatus, electronic devices, and storage media are basically similar to the system embodiments, so the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0231] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of protection of the present invention.

Claims

1. A sample feature generation system, characterized in that, The system includes: a management node, a reading node, and a processing node; wherein... The management node is used to obtain target operators from existing operators based on the function of the machine learning model to be trained, and determine the execution order of each target operator; select target nodes corresponding to each target operator from the processing nodes, deploy each target operator to the corresponding target node, and notify each target node of the path information of its downstream nodes based on the execution order. The reading node is used to obtain sample data without generated sample features from the external disk one by one, and send the obtained sample data to the first node each time. The first node is the target node corresponding to the first target operator in the execution order. The first node is used to store the received sample data in memory, process the sample data in memory using the deployed target operator to obtain a first result, and send the first result to its downstream nodes; The second node is used to store the first result sent by its upstream node into memory, and to process the first result in memory using the deployed target operator to obtain the second result. If the second node has a downstream node, it sends the second result to its downstream node. If the second node does not have a downstream node, it determines the second result as the sample feature of the sample data. The second node is any target node other than the first node.

2. The system according to claim 1, characterized in that, The management node, based on the function of the machine learning model to be trained, obtains target operators from existing operators and determines the execution order of each target operator, including: The management node determines candidate operators from existing operators based on the functionality of the machine learning model to be trained. Obtain relational data, wherein the operators corresponding to the relational objects recorded in the relational data are: the operators in the candidate operators, and the relations between the relational objects represent the execution order between the operators corresponding to the relational objects; The operator corresponding to the relation object is determined as the target operator, and the execution order of each target operator is determined according to the relationship between the relation objects.

3. The system according to claim 1, characterized in that, The management node is also configured to respond to an operator addition request, obtain a custom operator for processing sample data, and add the custom operator to the existing operators.

4. The system according to any one of claims 1-3, characterized in that, The system also includes: storage nodes with external disks mounted; The second node is used to send the determined sample features to the storage node in the absence of a downstream node; The storage node is used to store the received sample features to a mounted external disk.

5. The system according to any one of claims 1-3, characterized in that, The existing operators include at least one of the following operators: Data read operator, data write operator, data filter operator, data deduplication operator, data type conversion operator, data augmentation operator, and data merging operator.

6. A method for generating sample features, characterized in that, The target node is applied to the sample feature generation system, which includes a management node, a reading node, and a processing node. The target node is the node corresponding to each target operator selected by the management node from the processing node. The target operator is the operator obtained from existing operators according to the function of the machine learning model to be trained. The target node is deployed with corresponding target operators. The method includes: If the target node is the first node, it receives sample data without generated sample features obtained by the reading node from the external disk each time, stores the received sample data in memory, processes the sample data in memory using the deployed target operator to obtain a first result, and sends the first result to its downstream node so that the downstream node generates sample features of the sample data based on the first result. The first node is the target node corresponding to the first target operator in the execution order of the target operators determined by the management node, and the downstream node is the node determined according to the execution order. If the target node is a second node, the first result sent by its upstream node is stored in memory, and the first result in memory is processed by the deployed target operator to obtain the second result. If the second node has a downstream node, the second result is sent to its downstream node. If the second node does not have a downstream node, the second result is determined as the sample feature of the sample data. The second node is any target node other than the first node.

7. The method according to claim 6, characterized in that, The sample feature generation system also includes: a storage node mounted with an external disk; The method further includes: If the target node is a second node, the determined sample features are sent to the storage node in the absence of downstream nodes, so that the storage node stores the received sample features to the mounted external disk.

8. A method for generating sample features, characterized in that, The application sample feature generation system includes a management node, a reading node, and a processing node. The method includes: Based on the function of the machine learning model to be trained, the target operators are obtained from the existing operators, and the execution order of each target operator is determined. Select the target node corresponding to each target operator from the processing node, and deploy each target operator to the corresponding target node; Based on the execution order, the path information of its downstream nodes is notified to each target node, so that each target node generates sample features in the following manner: If the target node is a first node, it receives sample data without generated sample features obtained from the external disk by the reading node each time, stores the received sample data in memory, processes the sample data in memory using the deployed target operator to obtain a first result, and sends the first result to its downstream node so that the downstream node generates sample features of the sample data based on the first result. Here, the first node is the target node corresponding to the first target operator in the execution order, and the downstream node is the node determined according to the execution order. If the target node is a second node, it stores the first result sent by its upstream node in memory, processes the first result in memory using the deployed target operator to obtain a second result, sends the second result to its downstream node if the second node has a downstream node, and determines the second result as the sample feature of the sample data if the second node does not have a downstream node. Here, the second node is any target node other than the first node.

9. The method according to claim 8, characterized in that, The step of obtaining target operators from existing operators based on the function of the machine learning model to be trained, and determining the execution order of each target operator, includes: Based on the functionality of the machine learning model to be trained, candidate operators are determined from the existing operators; Obtain relational data, wherein the operators corresponding to the relational objects recorded in the relational data are: the operators in the candidate operators, and the relations between the relational objects represent the execution order between the operators corresponding to the relational objects; The operator corresponding to the relation object is determined as the target operator, and the execution order of each target operator is determined according to the relationship between the relation objects.

10. The method according to claim 8 or 9, characterized in that, The method further includes: In response to an operator addition request, a custom operator for processing the sample data is obtained, and the custom operator is added to the existing operators.

11. A sample feature generation device, characterized in that, The target node is applied to the sample feature generation system, which includes a management node, a reading node, and a processing node. The target node is the node corresponding to each target operator selected by the management node from the processing node. The target operator is the operator obtained from existing operators according to the function of the machine learning model to be trained. The target node is deployed with corresponding target operators. The device includes: The first feature generation module is configured to, if the target node is the first node, receive sample data without generated sample features obtained by the reading node from the external disk each time, store the received sample data in memory, process the sample data in memory using the deployed target operator to obtain a first result, and send the first result to its downstream node so that the downstream node generates sample features of the sample data based on the first result. The first node is the target node corresponding to the first target operator in the execution order of the target operators determined by the management node, and the downstream node is the node determined according to the execution order. The second feature generation module is used to store the first result sent by its upstream node into memory if the target node is a second node, and process the first result in memory using the deployed target operator to obtain the second result. If the second node has a downstream node, the module sends the second result to its downstream node. If the second node does not have a downstream node, the module determines the second result as the sample feature of the sample data. The second node is any target node other than the first node.

12. A sample feature generation device, characterized in that, The application sample feature generation system includes a management node, a reading node, and a processing node. The device includes: The operator acquisition module is used to obtain target operators from existing operators based on the functionality of the machine learning model to be trained, and to determine the execution order of each target operator. The node determination module selects the target node corresponding to each target operator from the processing nodes and deploys each target operator to the corresponding target node; The information notification module is used to notify each target node of the path information of its downstream nodes based on the execution order, so that each target node generates sample features in the following manner: If the target node is a first node, it receives sample data without generated sample features obtained from the external disk by the reading node each time, stores the received sample data in memory, processes the sample data in memory using the deployed target operator to obtain a first result, and sends the first result to its downstream node so that the downstream node generates sample features of the sample data based on the first result. Here, the first node is the target node corresponding to the first target operator in the execution order, and the downstream node is the node determined according to the execution order. If the target node is a second node, it stores the first result sent by its upstream node in memory, processes the first result in memory using the deployed target operator to obtain a second result, sends the second result to its downstream node if the second node has a downstream node, and determines the second result as the sample feature of the sample data if the second node does not have a downstream node. Here, the second node is any target node other than the first node.

13. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the steps of the method described in any one of claims 6-7 or 8-10.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the method described in any one of claims 6-7 or 8-10.

Images