A task scheduling method, device, storage medium and electronic device for a cloud platform

Abstract

An embodiment of the present invention discloses a task scheduling method, device, storage medium and electronic device for a cloud platform. The method includes: obtaining data to be processed, and respectively determining the similarity between the data to be processed and multiple prototype data; determining at least one target model from multiple student network models preset on the cloud platform based on the similarity, wherein each student network model is trained based on a similarity dataset corresponding to the prototype data; processing the data to be processed based on the at least one target model to obtain a processing result of the data to be processed. On the cloud platform, multiple student network models with small storage space and low computing power consumption are used to replace the processing model with large storage space and high computing power consumption, reducing the storage occupancy of the model on the cloud platform and the computing power consumed during the data processing process. At the same time, since the number of network parameters in the student network model is small and the calculation speed is fast, the processing efficiency of the data to be processed is improved.

Description

Technical Field

[0001] The embodiments of the present invention relate to the technical field of data processing, and in particular, to a task scheduling method, device, storage medium, and electronic device for a cloud platform. Background Art

[0002] The cloud platform is of great value to the popularization of artificial intelligence applications. On the one hand, artificial intelligence applications usually have a high dependence on computing hardware, and their costs are unaffordable for underdeveloped regions. The cloud platform can greatly reduce the requirements for local devices and make it easier to promote artificial intelligence applications. On the other hand, with the development of communication technology, the cloud platform can more easily achieve joint work between multiple centers and help industrial development.

[0003] However, since the development of artificial intelligence algorithms and the development of the cloud platform usually belong to two working teams, the cloud platform development team can only deploy the encapsulated model as a black box prediction model in the overall process of the cloud platform without the permission to modify the model, resulting in waste of cloud platform computing resources and reduction of efficiency. Summary of the Invention

[0004] The embodiments of the present invention provide a task scheduling method, device, storage medium, and electronic device for a cloud platform to achieve high-efficiency computing of the cloud platform.

[0005] In a first aspect, the embodiments of the present invention provide a task scheduling method for a cloud platform, including:

[0006] Obtain the data to be processed, and respectively determine the similarity between the data to be processed and multiple prototype data;

[0007] Based on the similarity, determine at least one target model among multiple student network models preset on the cloud platform, where each student network model is trained based on a similarity dataset corresponding to the prototype data;

[0008] Process the data to be processed based on the at least one target model to obtain the processing result of the data to be processed.

[0009] In a second aspect, the embodiments of the present invention further provide a task scheduling device for a cloud platform, including:

[0010] A data similarity determination module, configured to obtain the data to be processed and respectively determine the similarity between the data to be processed and multiple prototype data;

[0011] A target model determination module, configured to determine at least one target model among multiple student network models preset on the cloud platform based on the similarity, where each student network model is trained based on a similarity dataset corresponding to the prototype data;

[0012] The first data processing module is configured to process the data to be processed based on the at least one target model to obtain a processing result of the data to be processed.

[0013] In a third aspect, an embodiment of the present invention further provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the task scheduling method of the cloud platform provided in any embodiment of the present invention.

[0014] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the program is executed by a processor, it implements the task scheduling method of the cloud platform provided in any embodiment of the present invention.

[0015] The technical solution provided in this embodiment replaces a processing model with a large storage space and high computing power consumption with multiple student network models with a small storage space and low computing power consumption on the cloud platform, reducing the storage occupancy of the model on the cloud platform and the computing power consumed during the data processing process. At the same time, since the student network model has fewer network parameters and a fast calculation speed, the processing efficiency of the data to be processed is improved. For each data to be processed, one or more student network models are called as target models to process the data to be processed according to the similarity between the data to be processed and the prototype data corresponding to each student network model. On the basis of ensuring the processing accuracy of the data to be processed, the processing efficiency is improved, and the storage pressure and computing power pressure on the cloud platform are reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 is a schematic flowchart of a task scheduling method for a cloud platform provided by an embodiment of the present invention;

[0017] Figure 2 is a schematic structural diagram of an autoencoder network model provided by an embodiment of the present invention;

[0018] Figure 3 is a schematic diagram of the training process of a student network model provided by an embodiment of the present invention;

[0019] Figure 4 is a schematic flowchart of a processing process for data to be processed provided by an embodiment of the present invention;

[0020] Figure 5 is a schematic structural diagram of a task scheduling device for a cloud platform provided by an embodiment of the present invention;

[0021] Figure 6 is a schematic structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0022] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It can be understood that the specific embodiments described herein are only for explaining the present invention, rather than limiting the present invention. Additionally, it should be noted that for the convenience of description, only the parts related to the present invention rather than all the structures are shown in the accompanying drawings.

[0023] For any processing task, the cloud platform can configure a processing model, and for the data to be processed for the task to be executed, call the processing model to process the data to obtain processed data. Among them, the processing tasks can include, but are not limited to, image processing models, text processing models, and audio processing models, etc. Correspondingly, the processing models can be image processing models, text processing models, and audio processing models, etc. In some embodiments, the image processing tasks can include, but are not limited to, image classification tasks, image recognition tasks, image segmentation tasks, image super-resolution tasks, image style transfer tasks, image compression tasks, etc. Correspondingly, the image processing models include, but are not limited to, image classification models, image recognition models, image segmentation models, image super-resolution models, image style transfer models, image compression models, etc. The text processing tasks can include, but are not limited to, text classification tasks, abstract extraction tasks, text translation tasks, text keyword extraction tasks, etc. Correspondingly, the text processing models include, but are not limited to, text classification models, abstract extraction models, text translation models, text keyword extraction models, etc. The audio processing tasks include, but are not limited to, speech recognition tasks, audio noise reduction tasks, audio synthesis tasks, etc. Correspondingly, the audio processing models include, but are not limited to, speech recognition models, audio noise reduction models, audio synthesis models, etc.

[0024] Any of the above-mentioned processing models is obtained through iterative training with sample data and is capable of processing each data for the corresponding processing task. Taking the face recognition model as an example of the processing model, for the training samples of the face recognition model, including male face images, female face images, young face images, middle-aged face images, elderly face images, juvenile face images, baby face images, etc., the face recognition model obtained through training with the above sample images has the ability to perform face recognition on male face images, female face images, young face images, middle-aged face images, elderly face images, juvenile face images, baby face images. Correspondingly, during the training process, in order to improve the recognition accuracy and recognition range of the face recognition model, it is trained with a large number of different types of face sample images. The face recognition model obtained has many network parameters, occupies a large amount of storage resources, and consumes a large amount of computing power during the recognition process of any face image. Similarly, other image processing models, text processing models, and audio processing models are all obtained through the above method. Correspondingly, when any of the above processing models is applied on the cloud platform, the processing model occupies a large amount of storage resources, and each processing process consumes a large amount of computing power and takes a long operation time.

[0025] For the above technical problems, a task scheduling method for a cloud platform provided by an embodiment of the present invention is as follows. Refer to Figure 1 , Figure 1 which is a schematic flowchart of a task scheduling method for a cloud platform provided by an embodiment of the present invention. This embodiment is applicable to the situation of allocating tasks on a cloud platform. This method can be executed by a task scheduling device of the cloud platform provided by an embodiment of the present invention. The task scheduling device can be implemented by software and / or hardware, and the task scheduling device can be configured on an electronic computing device. The specific steps are as follows:

[0026] S110. Obtain the data to be processed, and respectively determine the similarity between the data to be processed and multiple prototype data.

[0027] S120. Based on the similarity, determine at least one target model from multiple student network models preset on the cloud platform, where each student network model is trained based on a similarity dataset corresponding to the prototype data.

[0028] S130. Process the data to be processed based on the at least one target model to obtain a processing result of the data to be processed.

[0029] In this embodiment, multiple student network models corresponding to any processing task are stored on the cloud platform. The number of student network models is at least two, and all are used to execute the same processing task. Taking the processing task of breast lesion recognition as an example, there are at least two corresponding student network models, all of which are used to identify lesions in breast images. Each of the above student network models is trained based on a similarity dataset of a prototype data, and the sample data corresponding to different student network models is different. Correspondingly, each student network model is used to process the similarity data of the corresponding prototype data.

[0030] By dividing the sample data of a processing task into multiple types and training a student network model based on the sample data of each type respectively, the similarity of the sample data is high and the sample data volume is small, which is convenient for the training process of the student network model to converge quickly, and the training speed is fast. At the same time, the above-mentioned quickly converging training process results in fewer network parameters of the student network model. Correspondingly, the computing power consumption of the student network model in each processing process is small.

[0031] Pre-store the prototype data corresponding to each student network model. The prototype data is determined according to the processing task of the student network model. Exemplarily, if the processing task of the student network model is face recognition, the prototype data is face images of different types; if the processing task of the student network model is breast lesion recognition, the prototype data is breast images of different types, and so on. It should be noted that there may be some overlapping samples in the sample data of different student network models, but they are not exactly the same.

[0032] For the data to be processed, determine the similarity between the data to be processed and each prototype data respectively to determine the prototype data corresponding to the data to be processed. Optionally, it can be by calculating the distance information between the data to be processed and each prototype data. For example, it can be calculating the Euclidean distance between the data to be processed and each prototype data, and using the distance information to represent the similarity between the data to be processed and each prototype data. The smaller the distance information between the data to be processed and the prototype data, the higher the similarity between the data to be processed and the prototype data. Optionally, when the data to be processed and the prototype data are image data, the similarity can be calculated based on the image content of the image, or based on the pixel points of each pixel point in the image. When the data to be processed and the prototype data are text data, the similarity can be calculated based on the key information in the text data; when the data to be processed and the prototype data are audio data, the text information in the audio data can be extracted to calculate the similarity, or the similarity can be calculated based on the feature information such as the pitch, timbre, and voiceprint of the audio data.

[0033] In some alternative embodiments, determining the similarity between the data to be processed and multiple prototype data respectively includes: extracting the feature information of the data to be processed based on a preset encoder; calculating the similarity between the extracted feature information and multiple prototype data in the prototype memory respectively.

[0034] In this embodiment, the prototype data can be the feature information extracted by a preset encoder. The determination method of the prototype data can be: performing feature extraction on a large amount of sample data to obtain the feature information, converting the feature information to the feature space, each feature information corresponds to a position information in the feature space respectively, performing clustering processing on the position information of each feature information to achieve the classification processing of a large amount of feature information in the feature space, and among the feature information corresponding to each class obtained by the clustering processing, determining the feature information corresponding to the class center as the prototype data corresponding to the class. Each class is respectively provided with a student network model for processing the data corresponding to the class.

[0035] Pre-store multiple prototype data, each prototype data representing a class. For the data to be processed, based on a preset encoder, feature extraction is performed on the data to be processed to obtain the feature information corresponding to the data to be processed. Calculate the similarity between the feature information and each prototype data respectively, and determine the class to which the data to be processed belongs based on the similarity. Among them, the similarity between the feature information and the prototype data can be determined by the Euclidean distance between the feature information and the prototype data, or can be determined by the distance between the feature information and the prototype data in the feature space. Specifically, when the similarity between the feature information of the data to be processed and any prototype data is greater than the similarity threshold, it is determined that the data to be processed belongs to the class where the prototype data is located. Among them, the similarity threshold can be determined in advance, for example, it can be 50%, and there is no limitation on this. It should be noted that the data to be processed can be such that the similarity with one or more prototype data satisfies the similarity threshold, that is, the data to be processed can belong to one or more classes simultaneously.

[0036] Based on the prototype data that meets the similarity condition, determine at least one target model for processing the data to be processed. Specifically, call the student network model corresponding to each prototype data that meets the similarity condition as the target model. Output the data to be processed to the target model respectively, and obtain the processing results output by each target model.

[0037] When there is one target model, determine the processing result of the target model as the processing result of the data to be processed; when the number of target results is two or more, determine the processing result of the data to be processed based on the processing results of multiple target models. Exemplarily, it can be to perform mean processing on the processing results of multiple target models to obtain the processing result of the data to be processed, or it can be to perform weighted processing on the processing results of multiple target models to obtain the processing result of the data to be processed, or it can be to process the processing results of multiple target models based on a preset result processing rule to obtain the processing result of the data to be processed. There is no limitation on this. Among them, the preset result processing rule can be determined based on the type of the processing result.

[0038] The technical solution provided in this embodiment replaces a processing model with a large storage space and high computing power consumption with multiple student network models with a small storage space and low computing power consumption on the cloud platform, reducing the storage occupancy of the model on the cloud platform and the computing power consumed during the data processing process. At the same time, since the number of network parameters in the student network model is small and the calculation speed is fast, the processing efficiency of the data to be processed is improved. For each data to be processed, based on the similarity between the data to be processed and the prototype data corresponding to each student network model, call one or more student network models as target models to process the data to be processed, improving the processing efficiency while ensuring the processing accuracy of the data to be processed, and reducing the storage pressure and computing power pressure on the cloud platform.

[0039] Based on the above embodiments, the encoder and the prototype memory are pre-trained. Specifically, the sample data of the encoder and the prototype memory can be a set of sample data of each student network model.

[0040] Optionally, the method for determining the encoder and the prototype memory includes: constructing an auto-encoding network model, training the auto-encoding network model based on preset sample data, and obtaining the encoder and the prototype memory when the auto-encoding network model meets the training conditions. Wherein, the auto-encoding network structure includes an encoder, a prototype addressing module, a decoder, and a prototype memory. The encoder is used to extract feature information of the input data, the prototype memory is used to store prototype data, the prototype addressing module is used to perform feature recombination based on similar prototype data to the input data, and the decoder is used to reconstruct data based on the recombined feature information.

[0041] Exemplarily, referring to Figure 2 , Figure 2 FIG. is a schematic structural diagram of an auto-encoding network model provided by an embodiment of the present invention. The encoder and the decoder in the auto-encoding network model can be neural network modules respectively, and can each include one or more convolutional blocks, and each convolutional block can include multiple convolutional layers. In some embodiments, the convolutional block can be a residual block. In some embodiments, the convolutional block further includes a pooling layer, an activation function layer, etc. arranged after each convolutional layer. It should be noted that the structures of the encoder and the decoder are not limited and can be set according to user requirements.

[0042] Perform self-supervised iterative training on the auto-encoding network model based on the sample data. Input the sample data into the auto-encoding network model. Specifically, the encoder extracts features from the input sample data to obtain feature information, inputs the feature information into the prototype memory, the prototype memory converts the feature information into the feature space, and determines the clustering process of the feature information with the stored feature information, determines the distance information between the feature information and the prototype data of each class, determines the class to which the feature information belongs through the distance information, and the similarity between the feature information and each prototype data. Wherein, the prototype memory is an external storage module with K prototype positions, and is used to save K prototype data in the training data. Optionally, the K prototype data can be the feature information at the class center position of each of the K classes obtained by clustering the feature information of the sample data during the training process. It should be noted that the number of prototype data stored in the prototype memory can be less than or equal to K, and during the training process, the prototype data stored in the prototype memory is updated with the training process to optimize the prototype data. Wherein, the optimization of the prototype data includes optimizing the prototype data corresponding to the class center of each class, and optimizing the number of prototype data.

[0043] The prototype addressing module determines similar prototype data based on the similarity between the feature information and each prototype data. Exemplarily, the similar prototype data can be a preset number (for example, three) of prototype data. For example, the prototype data are sorted based on the similarity between the feature information and each prototype data, and a preset number of prototype data are selected as the similar prototype data based on the sorting. Data fusion is performed on each similar prototype data to obtain the recombined feature information. Among them, performing data fusion on each similar prototype data can be to determine the weight of the corresponding prototype data based on the similarity between each prototype data and the characteristic information, and perform feature fusion on each similar prototype data based on the weight. Exemplarily, the similarities between each prototype data and the characteristic information are normalized to obtain the weights of the corresponding prototype data. In some alternative embodiments, the feature information and the prototype data can be in the form of matrices or vectors, which is not limited herein.

[0044] The decoder performs data reconstruction on the recombined feature. Exemplarily, taking the sample data as image data as an example, the decoder is used to reconstruct the recombined feature information into an output image. Taking the sample data as text data as an example, the decoder is used to reconstruct the recombined feature information into text data.

[0045] The input data of the autoencoding network model is used as the supervised data of the output data to implement the self-supervised training of the autoencoding network model, eliminating the need for manual setting of supervised data, reducing the data preprocessing process, and simplifying the training process. A loss function is generated, and the network parameters of the autoencoding network model are adjusted based on the loss function to implement the training of the autoencoding network model.

[0046] Among them, the loss function in the training process of the autoencoding network model includes: a reconstruction loss function based on the input data and the output data, and, when the distance difference between the input data and any two prototype data is less than a preset value, a prototype separation loss function is generated. The reconstruction loss function is a consistency loss function of the input data and the output data, and is determined based on the input data and the output data. Optionally, the reconstruction loss function can include, but is not limited to, a square loss function, an absolute value loss function, a cross-entropy cost function, a root mean square error, a mean square error, and a mean absolute error, which is not limited herein.

[0047] For the input data, determine the distance between the feature information of the input data and the prototype data stored in the prototype memory, and determine the distance difference between the input data and any two prototype data. The farther the distances between the prototype data in the prototype memory are from each other, the more representative the prototype data is, and there will be no conflict with the class regions of other prototype data. In this embodiment, the distance between the above prototype data is controlled by a prototype separation loss function. When the distance difference between the input data and any two prototype data is less than a preset value, a penalty is imposed on the autoencoder network model to widen the distance between the prototype data. Correspondingly, the prototype separation loss function is determined based on the distance difference between the above any two prototype data and a standard distance difference. The prototype separation loss function may include, but is not limited to, a squared loss function, a Hinge loss, a mean squared error, etc.

[0048] See Figure 2 , the reconstruction loss function is input backward from the output end of the decoder to the autoencoder network model, and the prototype separation loss function is input backward from the output end of the prototype memory to the autoencoder network model, so as to adjust the network parameters of the autoencoder network model. In some alternative embodiments, the adjustment of the network parameters of the autoencoder network model may be implemented based on the gradient descent method.

[0049] Iteratively execute the above training process. When the training conditions are met, determine that the training of the autoencoder network model is completed, and determine the encoder and the prototype memory in the trained autoencoder network model as the trained encoder and prototype memory for classifying the data to be processed. Among them, the training conditions may be one or more of a preset number of training times, a preset training accuracy, or the training process achieving convergence.

[0050] Based on the encoder and the prototype memory obtained by training in the above embodiment, classify the sample data, and train a student network model based on the sample data of each type. Exemplarily, see Figure 3 , Figure 3 is a schematic diagram of the training process of the student network model provided by the embodiment of the present invention.

[0051] Optionally, the training method of the student network model includes: obtaining a sample data set, dividing the sample data into multiple sample subsets based on the similarity between each sample data in the sample data set and each prototype data; for the sample data in each sample subset, determining the pseudo-labels of the sample data based on an original model, and training an initial network model based on the sample data in the sample subset and the corresponding pseudo-labels to obtain a student network model.

[0052] For the original data, i.e., the sample data in the sample dataset, the encoder extracts features from the original data, and determines the similarity between the extracted feature information and each prototype data in the prototype memory. When the similarity with any prototype data is greater than the similarity threshold, the prototype data is classified into the sample subset corresponding to the prototype data. It should be noted that when the similarity between any original data and multiple prototype data is greater than the similarity threshold, the original data can be classified into multiple sample subsets. Exemplarily, refer to Figure 3 the K data groups in. The original data is classified by the prototype data to form sample subsets corresponding to multiple classes, which are used to train different student network models respectively, reducing the diversity of the sample data of each student network model and facilitating the rapid training of the student network model to achieve convergence.

[0053] The pre-trained original model serves as the teacher network model, and this original model can process all types of data for the task, for example, it can be a pre-packaged processing model. For each sample data in each sample subset, the sample data is input into the original model, and the output result of the original model is used as the pseudo-label of the sample data. Based on the sample data in the sample subset and the corresponding pseudo-labels, a student network model is trained. Correspondingly, the sample data of each sample subset can be used to train the corresponding student network model respectively. Different student network models can be trained in parallel, which improves the training efficiency of the student network model.

[0054] Each of the above-trained student network models is used to process the data to be processed of its own type to obtain a processing result. Refer to Figure 4 , Figure 4 which is a schematic diagram of the processing flow of the data to be processed provided by an embodiment of the present invention. Figure 4Among them, for the data to be processed, that is, new data, feature information is extracted through the encoder, and the similarity with each prototype data is calculated in the prototype memory. In the case where there is only one prototype data exceeding the similarity threshold, the student network model corresponding to the matched prototype data is used as the target model, and the data to be processed is processed to obtain a processing result, that is, a prediction result. In the case where there are multiple prototype data exceeding the similarity threshold, the student network model corresponding to each prototype data exceeding the similarity threshold is determined as the target model respectively. The data to be processed is input into each target model to obtain a set of prediction results, and the final prediction result is determined based on the prediction results of each target model. In the case where there is no prototype data exceeding the similarity threshold, that is, the similarity between the data to be processed and multiple prototype data does not meet the matching conditions of each student network model, the data to be processed is processed based on the original model to obtain a processing result. Among them, the original model can be configured on the cloud platform and can be directly called. The original model can also be configured in the server. A processing request is generated based on the data to be processed, and the processing request is sent to the server so that the server calls the original model to process the data to be processed and feedback the processing result.

[0055] By setting the original model to process data beyond the capabilities of each student network model, the limitations of the student network model are avoided, and the processing accuracy of the data is improved.

[0056] Based on the above embodiments, the method further includes: storing the data to be processed processed by the original model in an independent data set. When the data in the independent data set meets the training conditions, new prototype data is determined based on the independent data set, and a new student network model is trained. Among them, the training conditions can be a preset training period or a preset data volume. The preset training period can be the update training period of the student network model. For example, it can be 1 day, 1 week or 1 month, etc. The preset data volume can be the cumulative data volume in the independent data set. For example, it can be 1000 or 2000, etc., and this is not limited.

[0057] By adding new data to the independent data set, the self-encoding network model is optimized with information to update the prototype data in the prototype memory. For example, new prototype data can be added, or the stored prototype data can be updated, etc. Based on the updated encoder and prototype memory, the sample data is divided, and the student network model is optimized based on the divided data. Exemplarily, in the case where new prototype data exists, a new student network model is trained based on the sample data of the class to which the new prototype data belongs; in the case where the stored prototype data changes, the corresponding student network model is optimized and trained based on the sample data of the class to which the changed prototype data belongs.

[0058] By independently storing newly added types of data and optimizing the student network model, the coverage of the student network model is improved, the learning ability and processing ability of the student network model are enhanced, and the processing accuracy of the data is increased.

[0059] Figure 5 FIG. 4 is a schematic structural diagram of a task scheduling device of a cloud platform provided by an embodiment of the present invention. The device includes:

[0060] A data similarity determination module 210, configured to obtain data to be processed and determine the similarity between the data to be processed and multiple prototype data respectively;

[0061] A target model determination module 220, configured to determine at least one target model from multiple student network models preset on the cloud platform based on the similarity, where each student network model is trained based on a similarity dataset corresponding to the prototype data;

[0062] A first data processing module 230, configured to process the data to be processed based on the at least one target model to obtain a processing result of the data to be processed.

[0063] Optionally, the data similarity determination module 210 is configured to:

[0064] Extract feature information of the data to be processed based on a preset encoder;

[0065] Calculate the similarity between the extracted feature information and multiple prototype data in the prototype memory respectively.

[0066] Optionally, the determination method of the encoder and the prototype memory includes:

[0067] Construct an autoencoder network model, where the autoencoder network structure includes an encoder, a prototype addressing module, a decoder, and a prototype memory. The encoder is used to extract feature information of the input data, the prototype memory is used to store prototype data, the prototype addressing module is used to perform feature recombination based on similar prototype data to the input data, and the decoder is used to perform data reconstruction based on the recombined feature information;

[0068] Train the autoencoder network model based on preset sample data, and when the autoencoder network model meets the training conditions, obtain the encoder and the prototype memory.

[0069] Optionally, the loss function in the training process of the autoencoder network model includes: a reconstruction loss function based on the input data and the output data, and a prototype separation loss function generated when the distance difference between the input data and any two prototype data is less than a preset value.

[0070] Optionally, the training method of the student network model includes:

[0071] Obtain a sample data set, and divide the sample data into multiple sample subsets based on the similarity between each sample data in the sample data set and each prototype data;

[0072] For the sample data in each sample subset, determine the pseudo-label of the sample data based on the original model, and train an initial network model based on the sample data in the sample subset and the corresponding pseudo-labels to obtain a student network model.

[0073] Optionally, an original model is also set on the cloud platform;

[0074] The device further includes:

[0075] A second data processing module, configured to, if the similarities between the data to be processed and multiple prototype data do not meet the matching conditions of each student network model, process the data to be processed based on the original model to obtain a processing result.

[0076] Optionally, the device further includes:

[0077] A model update module, configured to store the data to be processed processed by the original model in an independent data set, and based on the independent data set, determine new prototype data and train a new student network model when the data in the independent data set meets the training conditions.

[0078] The task scheduling device of the cloud platform provided by the embodiments of the present invention can execute the task scheduling method of the cloud platform provided by any embodiment of the present invention, and has corresponding functional modules and beneficial effects for executing the task scheduling method of the cloud platform.

[0079] Figure 6 It is a schematic structural diagram of an electronic device provided by an embodiment of the present invention. Figure 6 It shows a block diagram of an electronic device 12 suitable for implementing the embodiments of the present invention. Figure 6 The shown electronic device 12 is only an example, and should not bring any limitation to the functions and usage scope of the embodiments of the present invention. The device 12 is typically an electronic device that undertakes the function of image classification.

[0080] As Figure 6 shown, the electronic device 12 is presented in the form of a general computing device. The components of the electronic device 12 may include, but are not limited to: one or more processors 16, a storage device 28, and a bus 18 connecting different system components (including the storage device 28 and the processor 16).

[0081] Bus 18 represents one or more of several types of bus architectures, including a memory bus or memory controller, a peripheral bus, an Accelerated Graphics Port, a processor, or a local bus using any of the various bus architectures. By way of example, these architectures include, but are not limited to, Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MCA) bus, Enhanced ISA bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnect (PCI) bus.

[0082] Electronic device 12 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by electronic device 12, including volatile and nonvolatile media, removable and non-removable media.

[0083] The storage device 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. Electronic device 12 may further include other removable / non-removable, volatile / nonvolatile computer system storage media. By way of example only, storage system 34 can be used for reading and writing on non-removable, nonvolatile magnetic media ( Figure 6 not shown, typically referred to as a "hard disk drive"). Although Figure 6 not shown in the figure, a disk drive for reading and writing on a removable nonvolatile disk (such as a "floppy disk"), and an optical disk drive for reading and writing on a removable nonvolatile optical disk (such as a Compact Disc-Read Only Memory (CD-ROM), Digital Video Disc-Read Only Memory (DVD-ROM), or other optical media) can be provided. In these cases, each drive can be connected to bus 18 through one or more data media interfaces. The storage device 28 may include at least one program product having a set (e.g., at least one) of program modules that are configured to perform the functions of the embodiments of the present invention.

[0084] A program 36 having a set (at least one) of program modules 26 can be stored, for example, in a storage device 28. Such program modules 26 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include the implementation of a gateway environment. The program modules 26 generally execute the functions and / or methods in the embodiments described in the present invention.

[0085] The electronic device 12 can also communicate with one or more external devices 14 (such as a keyboard, a pointing device, a camera, a display 24, etc.), and can also communicate with one or more devices that enable a user to interact with the electronic device 12, and / or communicate with any device that enables the electronic device 12 to communicate with one or more other computing devices (such as a network card, a modem, etc.). Such communication can be carried out through an input / output (I / O) interface 22. Moreover, the electronic device 12 can also communicate with one or more gateways (such as a Local Area Network (LAN), a Wide Area Network (WAN), and / or a public gateway, such as the Internet) through a gateway adapter 20. As shown in the figure, the gateway adapter 20 communicates with other modules of the electronic device 12 through a bus 18. It should be understood that although not shown in the figure, other hardware and / or software modules can be used in combination with the electronic device 12, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, Redundant Arrays of Independent Disks (RAID) systems, tape drives, and data backup storage systems, etc.

[0086] The processor 16 executes various functional applications and data processing by running the program stored in the storage device 28, such as implementing the task scheduling method of the cloud platform provided in the above embodiments of the present invention.

[0087] An embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored. When the program is executed by a processor, it implements the task scheduling method of the cloud platform provided in the embodiments of the present invention.

[0088] Of course, the computer program stored on the computer-readable storage medium provided by the embodiments of the present invention is not limited to the method operations as described above, and can also execute the task scheduling method of the cloud platform provided by any embodiment of the present invention.

[0089] The computer storage medium of an embodiment of the present invention may adopt any combination of one or more computer-readable media. The computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples (non-exhaustive list) of the computer-readable storage medium include: an electrical connection having one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, the computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0090] The computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries computer-readable source code. Such a propagated data signal may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer-readable signal medium may also be any computer-readable medium other than the computer-readable storage medium, which can send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0091] The source code contained on the computer-readable medium may be transmitted by any appropriate medium, including but not limited to wireless, wire, optical fiber cable, RF, etc., or any suitable combination of the above.

[0092] The computer source code for performing the operations of the present invention may be written in one or more programming languages or combinations thereof. The programming languages include object-oriented programming languages such as Java, Smalltalk, C++, and also include conventional procedural programming languages such as the "C" language or similar programming languages. The source code may be executed entirely on the user's computer, partially on the user's computer, executed as an independent software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of gateway, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (for example, by using an Internet service provider to connect through the Internet).

[0093] Note that the above is only a preferred embodiment of the present invention and the technical principles applied. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein. Various obvious changes, re-adjustments, and substitutions can be made by those skilled in the art without departing from the protection scope of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments. Without departing from the concept of the present invention, more other equivalent embodiments can be included, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A task scheduling method for a cloud platform, characterized in that, Including: Obtain the data to be processed, and respectively determine the similarity between the data to be processed and multiple prototype data; Based on the similarity, determine at least one target model among multiple student network models preset on the cloud platform, where each student network model is respectively trained based on a similarity dataset corresponding to the prototype data; Process the data to be processed based on the at least one target model to obtain the processing result of the data to be processed; The step of respectively determining the similarity between the data to be processed and multiple prototype data includes: Extract the feature information of the data to be processed based on a preset encoder; Calculate the similarity between the extracted feature information and multiple prototype data in the prototype memory respectively; The determination method of the encoder and the prototype memory includes: Construct an auto-encoding network model, where the auto-encoding network model includes an encoder, a prototype addressing module, a decoder, and a prototype memory. The encoder is used to extract the feature information of the input data, the prototype memory is used to store prototype data, the prototype addressing module is used to perform feature recombination based on the similar prototype data of the input data, and the decoder is used to reconstruct the data based on the recombined feature information; Train the auto-encoding network model based on preset sample data. When the auto-encoding network model meets the training conditions, obtain the encoder and the prototype memory, where the sample data is a set of sample data of each student network model; during the training process, the prototype data stored in the prototype memory is updated with the training process to optimize the prototype data, where the optimization of the prototype data includes optimizing the prototype data corresponding to the class center of each class and optimizing the quantity of the prototype data; An original model is also set on the cloud platform; The method further includes: if the similarity between the data to be processed and multiple prototype data does not meet the matching conditions of each student network model, then process the data to be processed based on the original model to obtain a processing result; Store the data to be processed processed by the original model in an independent dataset. When the data in the independent dataset meets the training conditions, determine new prototype data based on the independent dataset and train a new student network model; The method further includes: In the case of the existence of the new prototype data, train the new student network model based on the sample data of the class to which the new prototype data belongs; In the case of the change of the stored prototype data, optimize and train the corresponding student network model based on the sample data of the class to which the changed prototype data belongs.

2. The method according to claim 1, wherein The loss function in the training process of the auto-encoding network model includes: a reconstruction loss function based on the input data and the output data, and, when the distance difference between the input data and any two prototype data is less than a preset value, generate a prototype separation loss function.

3. The method according to claim 1, characterized in that, The training method of the student network model includes: Obtain a sample dataset, and divide the sample data into multiple sample subsets based on the similarity between each sample data in the sample dataset and each prototype data; For the sample data in each subset of samples, pseudo-labels of the sample data are determined based on an original model, and an initial network model is trained based on the sample data and corresponding pseudo-labels in the sample subset to obtain a student network model.

4. A task scheduling device for a cloud platform, characterized in that, It includes: A data similarity determination module, configured to obtain data to be processed and determine the similarities between the data to be processed and multiple prototype data respectively; A target model determination module, configured to determine at least one target model from multiple student network models preset on a cloud platform based on the similarities, where each student network model is trained based on a similarity dataset corresponding to the prototype data; A first data processing module, configured to process the data to be processed based on the at least one target model to obtain a processing result of the data to be processed; The data similarity determination module is configured to extract feature information of the data to be processed based on a preset encoder; calculate the similarities between the extracted feature information and multiple prototype data in a prototype memory respectively; The determination method of the encoder and the prototype memory includes: Construct an auto-encoding network model, where the auto-encoding network model includes an encoder, a prototype addressing module, a decoder, and a prototype memory. The encoder is configured to extract feature information of input data, the prototype memory is configured to store prototype data, the prototype addressing module is configured to perform feature recombination based on similar prototype data to the input data, and the decoder is configured to perform data reconstruction based on the recombined feature information; Train the auto-encoding network model based on preset sample data. When the auto-encoding network model meets the training conditions, obtain the encoder and the prototype memory, where the sample data is a set of sample data of each student network model; during the training process, the prototype data stored in the prototype memory is updated along with the training process to optimize the prototype data, where the optimization of the prototype data includes optimizing the prototype data corresponding to the class centers of each class and optimizing the quantity of the prototype data; an original model is also set on the cloud platform; The apparatus further includes: A second data processing module, configured to, if the similarities between the data to be processed and multiple prototype data do not meet the matching conditions of each student network model, process the data to be processed based on the original model to obtain a processing result; A model update module, configured to store the data to be processed processed by the original model in an independent dataset, and based on the independent dataset, determine new prototype data and train a new student network model when the data in the independent dataset meets the training conditions; The apparatus is further configured to, in the presence of the new prototype data, train the new student network model based on the sample data of the class to which the new prototype data belongs; In the case where the stored prototype data changes, optimize and train the corresponding student network model based on the sample data of the class to which the changed prototype data belongs.

5. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the task scheduling method of the cloud platform as described in any one of claims 1-3.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by a processor, it implements the task scheduling method of the cloud platform as described in any one of claims 1-3.

Images