A data processing method and apparatus

Abstract

Embodiments of the present application disclose a data processing method and device. When a task needs to be processed, a first processing model and a second processing model are determined from a model set. A first shared model and a second shared model are determined based on a third processing model which is not selected from the model set. The first shared model is generated based on an output result of the first processing model and an output result of the third processing model, and the second shared model is generated based on an output result of the second processing model and the output result of the third processing model. The data to be processed is input into the first processing model and the second processing model to obtain a first output result and a second output result. The first output result is input into the first shared model to obtain a third output result, and the second output result is input into the second shared model to obtain a fourth output result. A target output result is determined according to the third output result and the fourth output result, and the target output result is determined as the output result of the third processing model.

Description

Technical Field

[0001] This application relates to the field of data processing technology, specifically to a data processing method and apparatus. Background Technology

[0002] With the continuous development of machine learning, more and more technical fields are using machine learning for data processing, thereby enabling the provision of services using models trained by machine learning. However, existing models are treated as independent service units, failing to fully explore the knowledge correlations between different model components. This results in the need to provide independent service models for different services, reducing the service capabilities of the models and increasing service costs. Summary of the Invention

[0003] In view of this, embodiments of this application provide a data processing method and apparatus to enable multi-model collaborative data processing and improve the service capabilities of the models.

[0004] To address the above problems, the technical solutions provided in this application are as follows:

[0005] In a first aspect of this application, a data processing method is provided, the method comprising:

[0006] Obtain a model set, which includes multiple processing models;

[0007] Based on the requirements corresponding to the tasks to be processed, a first processing model and a second processing model are determined from the set of models.

[0008] The data to be processed in the task to be processed is input into the first processing model and the second processing model respectively to obtain the first output result and the second output result;

[0009] Based on the third processing model that was not selected in the model set, a first shared model and a second shared model are determined. The first shared model is generated by training with the output of the first processing model as training data and the output of the third processing model as labels. The second shared model is generated by training with the output of the second processing model as training data and the output of the third processing model as labels. The third processing model is any model among the models that were not selected in the model set.

[0010] Input the first output result into the first shared model to obtain the third output result;

[0011] The second output result is input into the second shared model to obtain the fourth output result;

[0012] According to the third output result and the fourth output result, a target output result is obtained, and the target output result is determined as the output result of the third processing model.

[0013] Optionally, the method further comprises:

[0014] The first shared model and the second shared model are assigned a set of weights.

[0015] According to the first weight, the second weight, the third output result and the fourth output result, a target output result is obtained.

[0016] Optionally, the method further comprises:

[0017] The first shared model and the second shared model are assigned a set of weights.

[0018] Based on the assigned set of weights, the output result of the first shared model and the output result of the second shared model, a prediction result is determined.

[0019] According to the prediction result and the output result of the third processing model, it is determined whether the prediction result meets a preset condition.

[0020] If the preset condition is not met, a set of weights is re-assigned, and the above operations are performed.

[0021] If the preset condition is met, the weights corresponding to the first shared model and the second shared model are determined respectively.

[0022] Optionally, the training process of the first shared model and the second shared model comprises:

[0023] First training data is obtained, and the training data is input into the first processing model to obtain a first training result.

[0024] The first training data is input into the third processing model to obtain a second training result.

[0025] The first training result is used as input data and the second training result is used as a label to train a first initial shared model to obtain the first shared model.

[0026] Second training data is input into the second processing model to obtain a third training result.

[0027] The second training data is input into the third processing model to obtain a fourth training result.

[0028] Train the second initial shared model by taking the second training result as input data and the fourth training result as labels, to obtain the second shared model.

[0029] Optionally, the method further comprises:

[0030] Selecting a first source model from the model set, the first source model being any processing model in the model set;

[0031] Determining whether the resource used by the first source model in processing the task is less than or equal to a resource threshold value;

[0032] If yes, inputting test data into the first source model to obtain an output result;

[0033] Inputting the output result into the shared model corresponding to the first source model respectively, and obtaining an output result of each shared model;

[0034] Obtaining a prediction accuracy corresponding to each shared model according to a label corresponding to the test data and the output result of each shared model;

[0035] When the prediction accuracy corresponding to each shared model meets a preset condition, activating the first source model to process the task by using the first source model.

[0036] Optionally, the method further comprises:

[0037] When the prediction accuracy corresponding to each shared model does not meet the preset condition, continuing to select a second source model from the model set;

[0038] Determining whether the resource used by the first source model and the second source model in processing the task is less than or equal to the resource threshold value;

[0039] If yes, inputting the test data into the second source model to obtain an output result;

[0040] Inputting the output result into the shared model corresponding to the second source model respectively, and obtaining an output result of each shared model;

[0041] Obtaining a prediction accuracy corresponding to each shared model according to the label corresponding to the test data, the output result of each shared model corresponding to the first source model, and the output result of each shared model corresponding to the second source model;

[0042] When the prediction accuracy corresponding to each shared model meets a preset condition, activating the first source model and the second source model to process the task by using the first source model and the second source model.

[0043] Optionally, the resource threshold is a time delay or a display memory.

[0044] In a second aspect, a data processing apparatus is provided. The apparatus comprises:

[0045] A first obtaining unit, configured to obtain a model set, the model set comprising a plurality of processing models;

[0046] A determining unit, configured to determine a first processing model and a second processing model from the model set based on a requirement corresponding to a to-be-processed task;

[0047] A second obtaining unit, configured to input to-be-processed data in the to-be-processed task into the first processing model and the second processing model respectively, and obtain a first output result and a second output result;

[0048] The determining unit is configured to determine a first shared model and a second shared model based on a third processing model that is not selected from the model set, the first shared model being generated by training with the output result of the first processing model as training data and the output result of the third processing model as a label, the second shared model being generated by training with the output result of the second processing model as training data and the output result of the third processing model as a label, and the third processing model being any one of the models that are not selected from the model set;

[0049] The second obtaining unit is further configured to input the first output result into the first shared model, and obtain a third output result;

[0050] The second obtaining unit is further configured to input the second output result into the second shared model, and obtain a fourth output result;

[0051] The second obtaining unit is further configured to obtain a target output result according to the third output result and the fourth output result, and determine the target output result as the output result of the third processing model.

[0052] In a third aspect, a computer readable storage medium is provided. The computer readable storage medium stores instructions. When the instructions run on a device, the device performs the data processing method in the first aspect.

[0053] In a fourth aspect, a computer program product is provided. When the computer program product runs on a device, the device performs the data processing method in the first aspect.

[0054] Therefore, the embodiments of the present application have the following beneficial effects:

[0055] The embodiment of the present application can determine a first processing model and a second processing model from a model set based on the requirements corresponding to the to-be-processed task when the to-be-processed task needs to be processed by multiple models. The first shared model and the second shared model are determined based on a third processing model that is not selected from the model set. The first shared model is generated based on the output result of the first processing model and the output result of the third processing model, and the second shared model is generated based on the output result of the second processing model and the output result of the third processing model. The to-be-processed data is input into the first processing model and the second processing model to obtain a first output result and a second output result. The first output result is input into the first shared model to obtain a third output result, and the second output result is input into the second shared model to obtain a fourth output result. The target output result is determined according to the third output result and the fourth output result, and the target output result is determined as the output result of the third processing model. That is, the target output result is used to predict the output result of the third processing model when processing the to-be-processed data. It can be seen that, by sharing and fusing multiple models at the knowledge level, the correlation between models is fully mined, the service capability and efficiency of the model are greatly improved under the same computing resource limitation, the cost of model service is reduced, and the additional cost is extremely low, which does not interfere with the original model service. BRIEF DESCRIPTION OF DRAWINGS

[0056] Figure 1 A flowchart of a method for constructing a shared model provided by the embodiment of the present application;

[0057] Figure 2 A flowchart of a weight distribution method provided by the embodiment of the present application;

[0058] Figure 3 A flowchart of a data processing method provided by the embodiment of the present application;

[0059] Figure 4 A structure diagram of a data transmission device provided by the embodiment of the present application. DETAILED DESCRIPTION

[0060] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the embodiment of the present application will be further described in detail below with reference to the drawings and specific embodiments.

[0061] Reference Figure 1 The figure is a flowchart of a shared model construction method provided by the embodiment of the present application, as shown in Figure 1 The method comprises the following steps.

[0062] S101: Obtain a model set, which comprises multiple processing models.

[0063] For example, the model set includes processing model A, processing model B, processing model C, and processing model D.

[0064] S102: A target model is determined from the plurality of processing models, the target model being any one of the plurality of processing models.

[0065] S103: The first data is input into the target model to obtain a first processing result.

[0066] For example, the selected target model is processing model B, and the first data is input into processing model B to obtain a first processing result b.

[0067] S104: The first data is input into the first processing model to obtain a second processing result.

[0068] The first processing model is other processing model in the model set except the target model. For example, the target model is processing model B, and the first processing model can be processing model A, processing model C, and processing model D. The first data is input into processing model A to obtain a second processing result a, the first data is input into processing model C to obtain a second processing result c, and the first data is input into processing model D to obtain a second processing result d.

[0069] S105: A shared model is generated by training using the training data composed of the second processing result and the first processing result.

[0070] In the embodiment, to make each first processing model behave like the target model, that is, to construct a knowledge mapping from the first processing model (source model) to the target model, the second processing result is used as training data, the first processing result is used as a label, and an initial network model is trained to generate a shared model. The output result of the shared model can predict the output result of the target model.

[0071] For example, the first processing model is processing model A, the second processing result is a, and the input result of the target model B is b. The initial network model is trained using (a, b) to generate a shared model AB. In use, the to-be-processed data k is input into processing model A, the result output by processing model A is input into shared model AB, and an output result m is obtained. The to-be-processed data k is input into the target model B to obtain an output result n. The output result m and the output result n are similar, and the output result m can reflect the output result n. That is, in actual task processing, the target model can not be used for task processing, but the shared model can be used to predict the processing result of the target model.

[0072] After the shared model is generated, when the output result corresponding to the target model is needed, the output of the target model can be predicted using the first processing model and the shared model, without using the target model for data processing.

[0073] It can be understood that for any one of the processing models in the model set except the target model, the shared model corresponding to the processing model can be generated by training through S104-S105. At the same time, for any one of the processing models in the model set, it can be used as a target model to construct the knowledge mapping of other processing models to the target model. For example, the model set includes four processing models, which are processing model A, processing model B, processing model C and processing model D. When the target model is processing model B, the first processing model is model A, the first data x is input into model A to obtain the processing result a, and the first data x is input into model B to obtain the processing result b. Take (a, b) as a set of training data to train the initial network model, so that the shared model generated by training can learn the knowledge mapping based on a to obtain b, thereby obtaining the shared model AB. When the first processing model is model C, the first data x is input into model C to obtain the processing result c, and the first data x is input into model B to obtain the processing result b. Take (c, b) as a set of training data to train the initial network model, so that the shared model generated by training can learn the knowledge mapping based on c to obtain b, thereby obtaining the shared model CB. Based on the above training process, the shared model DB can also be obtained.

[0074] For example, the target model is processing model C, and the first processing model is model A. The first data x is input into model A to obtain the processing result a, and the first data y is input into model C to obtain the processing result c. Take (a, c) as a set of training data to train the initial network model, so that the shared model generated by training can learn the knowledge mapping based on a to obtain c, thereby obtaining the shared model AC. When the first processing model is model B, the first data y is input into model B to obtain the processing result b, and the first data y is input into model C to obtain the processing result c. Take (b, c) as a set of training data to train the initial network model, so that the shared model generated by training can learn the knowledge mapping based on b to obtain c, thereby obtaining the shared model BC. Based on the above training process, the shared model DC can also be obtained.

[0075] That is, the embodiment proposes a scheme for the source model (the first processing model) to imitate the processing behavior of the target model, to construct the knowledge mapping from the source model to the target model, and to generate the shared model. The shared model can be a fully connected neural network, the structure of which is adaptively determined according to the output layers of the source model and the target model, and a simple heuristic method is used to construct a hidden layer with twice the length of the target output and nonlinear activation, which is super lightweight and theoretically capable of modeling any mapping function.

[0076] Through sharing and fusing the multiple models at the knowledge level, the correlation between the models is fully mined, the service capacity and efficiency of the models are greatly improved under the same computing resource limitation, the cost of the model service is reduced, and the additional cost is extremely low, which does not interfere with the original model service.

[0077] The application is suitable for a task scenario with multiple model services, including: 1) a data analysis cloud platform, such as a company business data mining platform and a government video analysis platform; 2) an intelligent Internet of Things edge node, such as a home security integrated machine, a smart community integrated machine, a smart factory integrated machine, and the like; and 3) an intelligent terminal device, such as a visitor recognition built-in camera and a local photo analysis application of a smart phone.

[0078] As known from the above embodiments, based on a same target model, multiple knowledge mappings from different source models to the target model, that is, different shared models, can be constructed. In order to fuse the results output by different shared models, weights can be allocated to different shared models, and then the target model is predicted according to the allocated weights and the output results. How to allocate the weights will be described below, and the following embodiments take two shared models corresponding to a same target model as an example for description.

[0079] Referring to Figure 2 , which is a weight allocation method flowchart provided by the embodiments of the application, as shown in Figure 2 , the method can include the following steps.

[0080] S201: A set of weights is allocated to a first shared model and a second shared model.

[0081] In the embodiment, the set of weights can be allocated to the first shared model and the second shared model according to an allocation rule. The allocation rule can be random allocation or average allocation (for example, the weight is 0.5). The first shared model and the second shared model are generated based on a same target model and different source models. For example, the allocated set of weights is q1 and q2, wherein q1 is the weight corresponding to the first shared model, and q2 is the weight corresponding to the second shared model.

[0082] S202: A prediction result is determined based on the allocated set of weights, an output result of the first shared model, and an output result of the second shared model.

[0083] In this embodiment, the to-be-processed data is input into the source model corresponding to the first shared model, the result output by the source model is input into the first shared model, and the output result of the first shared model is obtained. Similarly, the to-be-processed data is input into the source model corresponding to the second shared model, the result output by the source model is input into the second shared model, and the output result of the second shared model is obtained. Then, the prediction result is obtained by weighting and according to the weight and the output result of each shared model. For example, the output result of the first shared model is b1, the output result of the second shared model is b2, and the prediction result is q1*b1+q2*b2.

[0084] S203: determining whether the prediction result meets a preset condition according to the prediction result and the output result of the third processing model; if the prediction result does not meet the preset condition, returning to S201; if the prediction result meets the preset condition, performing S204.

[0085] In this embodiment, the to-be-processed data is input into the third processing model (target model) to obtain an output result, and whether the prediction result meets a preset condition is determined according to the input result and the prediction result. If the prediction result meets the preset condition, it indicates that the prediction result can be used to predict the output result of the third processing model, and the allocated weight is reasonable, and then S204 is performed. If the prediction result does not meet the preset condition, it indicates that the prediction result cannot be used to predict the output result of the third processing model for the time being, and the allocated weight is unreasonable, and then S201-S203 are re-executed until the allocated weight can make the prediction result reflect the output result of the third processing model. The preset condition can be prediction accuracy or a distance threshold. The distance refers to the distance between the prediction result and the output result of the third processing model, which can be Mahalanobis distance or Euclidean distance.

[0086] S204: determining the weight corresponding to the first shared model and the weight corresponding to the second shared model, respectively.

[0087] For example, if it is determined that a set of weights meeting the preset condition is (0.8, 0.2), it is determined that the weight corresponding to the first shared model is 0.8 and the weight corresponding to the second shared model is 0.2.

[0088] In a specific embodiment, the weight corresponding to the first shared model and the weight corresponding to the second shared model can also be obtained by training according to the output result of the first shared model, the output result of the second shared model as input data, and the output result of the third processing model as a label.

[0089] Alternatively, the equation can be constructed to obtain the respective weights of the two shared models. For example, the weight corresponding to the first shared model is q1, the output result of the first shared model is b1; the weight corresponding to the second shared model is q2, the output result of the second shared model is b2, and the output result of the third processing model is b. According to q1*b1+q2*b2=b and q1+q2=1, the optimal solution is obtained, which is the weight corresponding to the two shared models.

[0090] It should be noted that when there are multiple shared models, the weight distribution can also be realized by the above method.

[0091] As can be seen from the above embodiments, for a target model, multiple shared models mapping to the knowledge of the target model can be trained based on different source models. For example, the model set includes four processing models (A, B, C, and D). When the target model is processing model B, shared model AB, shared model CB, and shared model DB can be trained, and the weights corresponding to the shared models are p1, p2, and p3. When the target model is processing model C, shared model AC, shared model BC, and shared model DC can be trained, and the weights corresponding to the shared models are q1, q2, and q3. When the target model is processing model D, shared model AD, shared model BD, and shared model CD can be trained, and the weights corresponding to the shared models are k1, k2, and k3. When the target model is processing model A, shared model BA, shared model CA, and shared model DA can be trained, and the weights corresponding to the shared models are r1, r2, and r3. That is, for any source model, shared models to multiple target models can be trained. For example, when the source model is processing model A, the shared models corresponding thereto can include shared model AB, shared model AC, and shared model AD. When the source model is processing model B, the shared models corresponding thereto can include shared model BA, shared model BC, and shared model BD.

[0092] After determining the shared model corresponding to each processing model, one or more processing models that can meet the demand can be selected from the plurality of processing models according to the demand corresponding to the processing task. Specifically, a first source model is selected from the plurality of processing models; it is judged whether the resource used when the first source model processes the task is less than the resource threshold value, if yes, test data is input into the first source model to obtain an output result; the output result is input into the shared model corresponding to the first source model respectively to obtain the output result of each shared model; the prediction accuracy corresponding to the shared model is obtained according to the label corresponding to the test data and the output result of the shared model; when the prediction accuracy does not meet the preset condition, a second source model is continuously selected from the model set; it is judged whether the resource used when the first source model and the second source model process the task is less than the resource threshold value, if yes, test data is input into the first source model and the second source model respectively to obtain an output result; the first source model output result is input into the shared model corresponding to the first source model and the second source model output result is input into the shared model corresponding to the second source model to obtain the output result of each shared model; the prediction accuracy of the shared model is obtained according to the label corresponding to the test data and the output result of the shared model; when the prediction accuracy does not meet the preset condition, a third source model is continuously selected from the model set and the above operation is continuously executed; if the prediction accuracy meets the preset condition, the selection is stopped. The resource threshold value can be the maximum delay or the maximum display memory corresponding to the processing task.

[0093] For example, the model set includes processing model A, processing model B, processing model C and processing model D, and the resource threshold value is the maximum processing delay of 40s. The processing model A is selected as the first source model for the first time, the processing delay of which is 20s, and the shared models corresponding thereto are model AB, model AC and model AD respectively. The test data x is obtained, input into the first source model A to obtain the output result a, and the output result is input into the model AB, the model AC and the model AD respectively to obtain the output results b', c' and d'. The prediction accuracy of the model AB is obtained according to the label b corresponding to the test data x and b', the prediction accuracy of the model AC is obtained according to the label c corresponding to the test data x and c', and the prediction accuracy of the model AD is obtained according to the label d corresponding to the test data x and d'. When each of the above prediction accuracies meets the preset condition, it is not necessary to select from the model set again; if the above prediction accuracies do not meet the preset condition, the selection from the model set is continuously performed.

[0094] For example, the second selection processing model B is a second source model, and the processing time delay of the second source model is 15s. The sum of the processing time delays of the first source model and the second source model is less than the resource threshold 40s. The test data x is input into the first source model A to obtain an output result a. The output result is input into the model AC and the model AD respectively to obtain output results c1' and d1'. The test data x is input into the second source model B to obtain an output result b. The output result is input into the model BC and the model BD respectively to obtain output results c2' and d2'. According to the weight q1 corresponding to the model AC and the weight q2 corresponding to the model BC, c' = q1*c1' + q2*c2' is obtained. According to the weight k1 corresponding to the AD model and the weight k2 corresponding to the model BD, d' = k1*d1' + k2*d2' is obtained. According to the label c corresponding to the test data x and c', the prediction accuracy corresponding to the model AC+model BC is obtained. According to the label d corresponding to the test data x and d', the prediction accuracy corresponding to the model AD+model BD is obtained. When each of the above prediction accuracies meets the preset condition, it is not necessary to select again from the model set. If the above prediction accuracies do not meet the preset condition, a third source model is selected from the model set until the resources used by the selected multiple models for processing the task are greater than the resource threshold or the prediction accuracy meets the preset condition.

[0095] Based on the above method, the combination of models used for processing a task can be determined. For the convenience of understanding, the following will be described with reference to the accompanying drawings.

[0096] Referring to Figure 3 The figure is a data processing method flowchart provided by an embodiment of the present application, as Figure 3 indicated, the method can include:

[0097] S301: Obtain a model set, which includes multiple processing models.

[0098] For example, the model set includes processing model A, processing model B, processing model C, and processing model D.

[0099] S302: Based on the requirements of a to-be-processed task, determine a first processing model and a second processing model from the model set.

[0100] In the embodiment, when a model is needed for task processing, part of the processing models can be selected from the multiple processing models based on the requirements corresponding to the to-be-processed task to execute the task. The requirements corresponding to the to-be-processed task can include the maximum time delay requirement that needs to be met during processing. For example, the task requirements can be met by using the processing model A and the processing model C in the model set. The first processing model can be the processing model A, and the second processing model can be the processing model C.

[0101] S303: input the to-be-processed data in the to-be-processed task into the first processing model and the second processing model respectively to obtain a first output result and a second output result.

[0102] For example, the to-be-processed data is z, z is input into the processing model A as input data to obtain the first output result a, and z is input into the processing model C as input data to obtain the second output result c.

[0103] S304: determining a first shared model and a second shared model based on the third processing model that is not selected.

[0104] The first shared model is generated by training the output result of the first processing model as training data and the output result of the third processing model as a label, the second shared model is generated by training the output result of the second processing model as training data and the output result of the third processing model as a label, and the third processing model is any one of the models in the model set that is not selected.

[0105] For example, when the third processing model is model B, the first shared model is AB and the second shared model is CB. When the third processing model is D, the first shared model can be AD and the second shared model can be CD.

[0106] It should be noted that the execution order of S303 and S304 in the embodiment is not limited by the above, and S304 can be executed first and then S303, or S303 and S304 can be executed simultaneously.

[0107] S305: input the first output result into the first shared model to obtain a third output result.

[0108] S306: input the second output result into the second shared model to obtain a fourth output result.

[0109] S307: obtain a target output result according to the third output result and the fourth output result, and determine the target output result as the output result of the third processing model.

[0110] Specifically, as known from the above embodiment, a weight can be assigned to each shared model, and when the target output result is determined, a first weight corresponding to the first shared model and a second weight corresponding to the second shared model can be obtained; the target output result is obtained according to the first weight, the second weight, the third output result and the fourth output result, and the target output result is determined as the output result of the third processing model. That is, the target output result is used to predict the output result of the third processing model.

[0111] It can be seen that by sharing multiple models at the knowledge level and fully mining the relevance between the models, the service capability and efficiency of the models are greatly improved and the cost of model service is reduced under the same computing resource limitation. Meanwhile, the original model server is not disturbed.

[0112] Based on the method embodiments, the embodiments of the present application further provide a data processing apparatus, which will be described below with reference to the accompanying drawings.

[0113] Referring to Figure 4 , the figure is a data processing apparatus structure diagram provided by the embodiments of the present application, as Figure 4 shown, the apparatus 400 comprises:

[0114] The first acquisition unit 401 is configured to acquire a model set, wherein the model set comprises a plurality of processing models;

[0115] The determination unit 402 is configured to determine a first processing model and a second processing model from the model set based on the requirements corresponding to the to-be-processed task;

[0116] The second acquisition unit 403 is configured to input the to-be-processed data in the to-be-processed task into the first processing model and the second processing model respectively, and obtain a first output result and a second output result;

[0117] The determination unit 402 is configured to determine a first shared model and a second shared model based on a third processing model that is not selected from the model set, the first shared model is generated by training the output result of the first processing model as training data and the output result of the third processing model as a label, the second shared model is generated by training the output result of the second processing model as training data and the output result of the third processing model as a label, and the third processing model is any one of the models that are not selected from the model set;

[0118] The second acquisition unit 403 is configured to input the first output result into the first shared model and obtain a third output result;

[0119] The second acquisition unit 403 is configured to input the second output result into the second shared model and obtain a fourth output result;

[0120] The second acquisition unit 403 is configured to acquire a target output result according to the third output result and the fourth output result, and determine the target output result as the output result of the third processing model.

[0121] In a specific implementation, the second obtaining unit is specifically configured to obtain a first weight corresponding to the first shared model and a second weight corresponding to the second shared model; and obtain a target output result according to the first weight, the second weight, the third output result and the fourth output result.

[0122] In a specific implementation, the apparatus further includes an allocating unit, which is specifically configured to allocate a set of weights to the first shared model and the second shared model; determine a prediction result based on the allocated set of weights, an output result of the first shared model and an output result of the second shared model; determine whether the prediction result meets a preset condition according to the prediction result and an output result of the third processing model; if the prediction result does not meet the preset condition, re-allocate a set of weights and perform the above operations; and if the prediction result meets the preset condition, determine the weights corresponding to the first shared model and the second shared model respectively.

[0123] In a specific implementation, the training process of the first shared model and the second shared model includes: obtaining first training data, inputting the training data into the first processing model to obtain a first training result; inputting the first training data into the third processing model to obtain a second training result; training a first initial shared model by taking the first training result as input data and the second training result as a label to obtain the first shared model; inputting second training data into the second processing model to obtain a third training result; inputting the second training data into the third processing model to obtain a fourth training result; training a second initial shared model by taking the second training result as input data and the fourth training result as a label to obtain the second shared model.

[0124] In a specific implementation, the apparatus further includes:

[0125] The selecting unit is configured to select a first source model from the model set, the first source model being any processing model in the model set;

[0126] The judging unit is configured to judge whether a resource used by the first source model in processing the task is less than or equal to a resource threshold value;

[0127] The second obtaining unit is further configured to input test data into the first source model to obtain an output result when the judging unit judges that the answer is yes.

[0128] The second obtaining unit is further configured to input the output result into a shared model corresponding to the first source model respectively, and obtain an output result of each shared model.

[0129] The third obtaining unit is configured to obtain a prediction accuracy corresponding to each shared model according to a label corresponding to the test data and an output result of each shared model.

[0130] The activation unit is configured to activate the first source model to process the task by using the first source model when the prediction accuracy corresponding to each shared model meets a preset condition.

[0131] In a specific implementation manner,

[0132] The selection unit is further configured to continue to select a second source model from the model set when the prediction accuracy corresponding to each shared model does not meet the preset condition.

[0133] The judgment unit is configured to judge whether resources used by the first source model and the second source model to process the task are less than or equal to the resource threshold.

[0134] The second obtaining unit is configured to input the test data into the second source model to obtain an output result when the judgment result of the judgment unit is yes.

[0135] The second obtaining unit is configured to input the output result into each shared model corresponding to the second source model respectively, and obtain an output result of each shared model.

[0136] The third obtaining unit is further configured to obtain a prediction accuracy corresponding to each shared model according to a label corresponding to the test data, an output result of each shared model corresponding to the first source model, and an output result of each shared model corresponding to the second source model.

[0137] The activation unit is further configured to activate the first source model and the second source model to process the task by using the first source model and the second source model when the prediction accuracy corresponding to each shared model meets a preset condition.

[0138] It should be noted that the implementation of each unit in this embodiment can refer to the related description in the method embodiment shown in Figures 1-3 This embodiment will not be described herein again.

[0139] In addition, the embodiment of the present application further provides a computer readable storage medium, the computer readable storage medium stores instructions, when the instructions run on the device, make the device execute the data processing method.

[0140] The embodiment of the present application provides a computer program product, when the computer program product runs on the device, makes the device execute the data processing method.

[0141] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems or apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple, and relevant parts can be referred to the method section.

[0142] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0143] It should also be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0144] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0145] The foregoing description of the disclosed embodiments enables a person skilled in the art to make or use the application. Modifications of these embodiments will occur to persons of skill in the art, and that the generic principles defined herein can be applied to other embodiments without departing from the spirit or scope of the application. Therefore, the present application is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A data processing method, characterized by, The method comprises: acquiring a model set comprising a plurality of processing models; determining a first processing model and a second processing model from the model set based on a requirement corresponding to a to-be-processed task; the requirement corresponding to the to-be-processed task comprises a delay requirement to be met during processing; inputting to-be-processed data in the to-be-processed task into the first processing model and the second processing model respectively to obtain a first output result and a second output result; determining a first shared model and a second shared model based on a third processing model that is not selected from the model set; the first shared model is generated by training with the output result of the first processing model as training data and the output result of the third processing model as a label; the second shared model is generated by training with the output result of the second processing model as training data and the output result of the third processing model as a label; the third processing model is any one of the models that are not selected from the model set; inputting the first output result into the first shared model to obtain a third output result; inputting the second output result into the second shared model to obtain a fourth output result; acquiring a target output result according to the third output result and the fourth output result, and determining the target output result as the output result of the third processing model; the method is used for a data analysis cloud platform, an intelligent Internet of Things edge node, and an intelligent terminal device; a training process of the first shared model and the second shared model comprises: acquiring first training data, inputting the training data into the first processing model to obtain a first training result; inputting the first training data into the third processing model to obtain a second training result; training a first initial shared model with the first training result as input data and the second training result as a label to obtain the first shared model; and inputting second training data into the second processing model to obtain a third training result; inputting the second training data into the third processing model to obtain a fourth training result; training a second initial shared model with the second training result as input data and the fourth training result as a label to obtain the second shared model; the method further comprises: selecting a first source model from the model set; the first source model is any processing model in the model set; judging whether a resource used by the first source model when processing a task is less than or equal to a resource threshold value; if yes, inputting test data into the first source model to obtain an output result; inputting the output result into shared models corresponding to the first source model respectively to obtain output results of the shared models; acquiring a prediction accuracy corresponding to each shared model according to a label corresponding to the test data and the output results of the shared models; when the prediction accuracy corresponding to each shared model meets a preset condition, activating the first source model to process a task by using the first source model; the resource threshold value is maximum display memory.

2. The method of claim 1, wherein, The method further comprises: allocating a set of weights for the first shared model and the second shared model; determining a prediction result based on the allocated set of weights, an output result of the first shared model, and an output result of the second shared model; 3. The method of claim 2, wherein, determining whether the prediction result meets a preset condition according to the prediction result and an output result of the third processing model; if the preset condition is not met, re-allocating a set of weights and performing the above operations; if the preset condition is met, determining the weights corresponding to the first shared model and the second shared model respectively. The method further comprises: when the prediction accuracy corresponding to each shared model does not meet the preset condition, continuing to select a second source model from the model set; determining whether the resources used by the first source model and the second source model when processing the task are less than or equal to the resource threshold value; 4. The method of claim 1, wherein, if yes, inputting the test data into the second source model to obtain an output result; inputting the output result into the shared model corresponding to the second source model respectively, and obtaining an output result of each shared model; obtaining the prediction accuracy corresponding to each shared model according to the label corresponding to the test data, the output result of each shared model corresponding to the first source model, and the output result of each shared model corresponding to the second source model; when the prediction accuracy corresponding to each shared model meets the preset condition, activating the first source model and the second source model to process the task by using the first source model and the second source model. The device comprises: a first obtaining unit configured to obtain a model set, the model set comprising a plurality of processing models; a determining unit configured to determine a first processing model and a second processing model from the model set based on a requirement corresponding to a to-be-processed task, the requirement including a delay requirement to be met when processing; 5. A data processing apparatus, characterized by, a second obtaining unit configured to input to-be-processed data in the to-be-processed task into the first processing model and the second processing model respectively to obtain a first output result and a second output result; the determining unit is configured to determine a first shared model and a second shared model based on a third processing model that is not selected from the model set, the first shared model being generated by training with the output result of the first processing model as training data and the output result of the third processing model as a label, the second shared model being generated by training with the output result of the second processing model as training data and the output result of the third processing model as a label, and the third processing model being any one of the models that are not selected from the model set. ​ ​ ​ The second obtaining unit is further configured to input the first output result into the first shared model to obtain a third output result. The second obtaining unit is further configured to input the second output result into the second shared model to obtain a fourth output result. The second obtaining unit is further configured to obtain a target output result according to the third output result and the fourth output result, and determine the target output result as the output result of the third processing model. The device is used for a data analysis cloud platform, an intelligent Internet of Things edge node, and an intelligent terminal device. The training process of the first shared model and the second shared model includes: obtaining first training data, inputting the training data into the first processing model to obtain a first training result; inputting the first training data into a third processing model to obtain a second training result; training a first initial shared model by taking the first training result as input data and the second training result as a label to obtain the first shared model; and inputting second training data into the second processing model to obtain a third training result; inputting the second training data into a third processing model to obtain a fourth training result; training a second initial shared model by taking the second training result as input data and the fourth training result as a label to obtain the second shared model. The selecting unit is configured to select a first source model from the model set, the first source model being any processing model in the model set. The judging unit is configured to judge whether a resource used by the first source model in processing a task is less than or equal to a resource threshold value. The second obtaining unit is further configured to input test data into the first source model to obtain an output result when the judging unit judges that the resource used by the first source model in processing the task is less than or equal to the resource threshold value. The second obtaining unit is further configured to input the output result into a shared model corresponding to the first source model to obtain an output result of each shared model. The third obtaining unit is configured to obtain a prediction accuracy corresponding to each shared model according to a label corresponding to the test data and the output result of each shared model. The activating unit is configured to activate the first source model to process the task by using the first source model when the prediction accuracy corresponding to each shared model meets a preset condition. The resource threshold value is maximum display memory.

6. A computer-readable storage medium, characterized in that, The computer readable storage medium stores instructions, and when the instructions run on the device, the device executes the data processing method in any one of claims 1-4.

7. A computer program product, characterised in that, The computer program product runs on the device, and the device executes the data processing method in any one of claims 1-4.

Images