A model production method and device, electronic equipment and storage medium

Abstract

The application provides a model production method and device, electronic equipment and a storage medium. The method comprises the following steps: in response to a data access request, accessing training data from a data source and processing the training data to obtain a target data set; in response to a model access request, selecting a target model matched with the access request from a model warehouse, wherein the model access request carries model type information and model accuracy information; in response to a resource scheduling request, determining a target resource through resource specification information, wherein the resource specification information is determined based on the target data set and the target model; and in response to a model training request, executing a training task based on the target resource, wherein the training task is created based on the target data set and the target model. The application can decouple each stage of model production, each stage can complete its own function, and the scalability, robustness, reusability and portability of model production are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and in particular to a model production method and device, electronic equipment and storage medium. BACKGROUND

[0002] With the development of artificial intelligence entering a period of vigorous development, artificial intelligence technology has been applied to various industries. Model is the core of artificial intelligence and is the fundamental way to make computers intelligent. Its application is widespread in various fields of artificial intelligence. In terms of practical significance, model production is a process of training a model using data, algorithms and training frameworks, and then using the model for inference application.

[0003] The traditional model production construction method is a purely manual method. Multiple scripts are written to process raw data to obtain a training data set of the model, model training code is written to train the model, and finally a model inference script is written to deploy the online model. The traditional model production line construction method requires manual configuration of dependent environments, manual running of scripts and collection of running results, manual deployment of models and maintenance of model services, which makes the model development cycle long, and the various steps of the model production line are difficult to upgrade and transform due to strong coupling, and have poor reusability. The manual environment configuration method also causes environment dependency conflicts and other problems. The traditional model production construction method is difficult to adapt to the rapid iteration needs of models caused by changes in the industry. SUMMARY

[0004] Therefore, the embodiments of the present application provide a model production method and device, electronic equipment and storage medium, which can decouple each stage of model production, complete their own functions in each stage, improve the scalability, robustness, reusability and portability of model production.

[0005] The technical solutions of the embodiments of the present application are as follows:

[0006] In a first aspect, the embodiments of the present application provide a model production method, which comprises:

[0007] In response to a data access request, training data is accessed from a data source, and the training data is processed to obtain a target data set;

[0008] In response to a model access request, a target model matched with the access request is selected from a model warehouse, wherein the access request carries model type information and model accuracy information;

[0009] In response to a resource scheduling request, a target resource is determined through resource specification information, wherein the resource specification information is determined based on the target data set and the target model;

[0010] In response to a model training request, a training task is executed based on the target resource, wherein the training task is created based on the target data set and the target model.

[0011] In a possible implementation, the method further includes:

[0012] In response to an inference service request, the trained model is published, and the running of the trained model is monitored to dynamically adjust the running configuration of the trained model.

[0013] In addition, the running information of the trained model is fed back to use the running information as a data source for optimizing the trained model.

[0014] In a second aspect, the embodiments of the present application further provide a model production device, and the device includes:

[0015] A data middle platform module is configured to access training data from a data source in response to a data access request, and process the training data to obtain a target data set.

[0016] A model warehouse module is configured to select a target model matched with the access request from a model warehouse in response to a model access request, wherein the model access request carries model type information and model precision information.

[0017] A resource scheduling module is configured to determine a target resource through resource specification information in response to a resource scheduling request, wherein the resource specification information is determined based on the target data set and the target model.

[0018] A training task module is configured to execute a training task based on the target resource in response to a model training request, wherein the training task is created based on the target data set and the target model.

[0019] In a possible implementation, the device further includes:

[0020] An inference service module is configured to publish a trained model and monitor the running of the trained model in response to an inference service request, so as to dynamically adjust the running configuration of the trained model.

[0021] In addition, the running information of the trained model is fed back to use the running information as a data source for optimizing the trained model.

[0022] In a third aspect, the embodiments of the present application further provide an electronic device, comprising a processor, a storage medium and a bus, the storage medium storing machine readable instructions executable by the processor, when the electronic device is running, the processor and the storage medium communicate through the bus, the processor executes the machine readable instructions to perform the model production method of any one of the first aspect.

[0023] In a fourth aspect, the embodiments of the present application further provide a computer readable storage medium, the computer readable storage medium storing a computer program, when the computer program is run by a processor, the model production method of any one of the first aspect is executed.

[0024] The embodiments of the present application have the following beneficial effects:

[0025] By decoupling each stage of model production, each stage can complete its own function and provide module capabilities to the outside. It can not only ensure the independent operation of each stage, but also enhance the operation efficiency of the whole system. The system expansion, robustness, reusability and portability are well supported. In addition to self-circulation, it can also ensure the gradual optimization and upgrading of the model, and can integrate the existing capabilities of third parties. Not only can the achievements of third parties be used, but also more suitable models can be produced according to different business scenarios based on the capabilities of third parties. BRIEF DESCRIPTION OF DRAWINGS

[0026] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed in the embodiments. It should be understood that the following drawings only show some embodiments of the present application, and therefore should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can also be obtained without creative labor.

[0027] Figure 1 is a flowchart of steps S101-S104 provided by the embodiments of the present application;

[0028] Figure 2 is a flowchart of steps S201-S202 provided by the embodiments of the present application;

[0029] Figure 3 is a structural diagram of a model production device provided by the embodiments of the present application;

[0030] Figure 4 is a structural diagram of an electronic device provided by the embodiments of the present application. DETAILED DESCRIPTION

[0031] In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. It should be understood that the drawings in the present application are only used for the purpose of description and illustration, and do not limit the scope of the present application. In addition, it should be understood that the schematic drawings are not drawn according to the actual proportions. The flowcharts used in the present application show the operations implemented according to some embodiments of the present application. It should be understood that the operations of the flowcharts can not be implemented in sequence, and the steps without logical context relationship can be reversed in sequence or implemented simultaneously. In addition, one or more other operations can be added to the flowcharts or one or more operations can be removed from the flowcharts under the guidance of the content of the present application.

[0032] In the following description, "some embodiments" are related to a subset of all possible embodiments, but it can be understood that "some embodiments" can be the same subset or different subsets of all possible embodiments, and can be combined with each other without conflict.

[0033] In addition, the described embodiments are only some of the embodiments of the present application, not all. The components of the embodiments of the present application described and shown in the drawings herein can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application. All other embodiments obtained by those skilled in the art based on the embodiments of the present application without creative labor are within the scope of protection of the present application.

[0034] In the following description, the term "first\second\third" is only to distinguish similar objects, and does not represent a specific order of the objects. It can be understood that "first\second\third" can be interchanged in a specific order or sequence as allowed, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0035] It should be noted that the term "comprising" will be used in the embodiments of the present application to indicate the presence of the features declared thereafter, but does not exclude the addition of other features.

[0036] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. The terms used herein are for the purpose of describing the embodiments of the present application, not for limiting the present application.

[0037] Reference is made to Figure 1 , Figure 1is a flowchart of the model production method steps S101-S104 provided by the embodiment of the present application, which will be described in combination with Figure 1 The steps S101-S104 shown will be described.

[0038] Step S101, in response to a data access request, accessing training data from a data source, and processing the training data to obtain a target data set;

[0039] Step S102, in response to a model access request, selecting a target model matching the access request from a model warehouse, wherein the access request carries model type information and model accuracy information;

[0040] Step S103, in response to a resource scheduling request, determining a target resource through resource specification information, wherein the resource specification information is determined based on the target data set and the target model;

[0041] Step S104, in response to a model training request, executing a training task based on the target resource, wherein the training task is created based on the target data set and the target model.

[0042] The above model production method decouples each stage of model production, achieves the function of each stage, and provides module capability. It can not only ensure the independent operation of each stage, but also enhance the operation efficiency of the whole system, and will not change the whole system due to the change of a certain stage. It has good support for system scalability, robustness, reusability, and portability. In addition to self-circulation, it can also ensure the gradual optimization and upgrading of the model, and can integrate the existing capabilities of third parties. Not only can the achievements of third parties be used, but also more needed models can be produced based on the capabilities of third parties according to different business scenarios.

[0043] The above exemplary steps of the embodiment of the present application will be described below.

[0044] In step S101, in response to a data access request, training data is accessed from a data source, and the training data is processed to obtain a target data set.

[0045] The data center module can execute this step, which mainly integrates various model required data, including external import, internal model inference generated data, and then processes and arranges the data according to the data specification required by the model, to provide usable data specification for other modules to reference according to the specification.

[0046] In step S102, in response to a model access request, a target model matching the access request is selected from a model warehouse, wherein the access request carries model type information and model accuracy information.

[0047] The model warehouse module can execute this step, mainly to integrate existing third-party models and manage the models generated by model training, control the models in various versions, and thus meet the needs of different business scenarios for the same model. The model warehouse module provides basic services for model training and model inference.

[0048] In step S103, in response to a resource scheduling request, a target resource is determined based on resource specification information, wherein the resource specification information is determined based on the target data set and the target model.

[0049] The resource scheduling module can execute this step, mainly to complete the allocation of resources, and allocate resources according to actual needs based on existing machine resources, including CPU, memory, GPU, and storage. The resources are not used unlimitedly, and the storage and calculation are used under the allocated resources. The resource specification required for use of the resources can be called to realize the scheduling and use of the resources.

[0050] In step S104, in response to a model training request, a training task is executed based on the target resource, wherein the training task is created based on the target data set and the target model.

[0051] The training task module can execute this step, mainly to complete the training of various models, and combine the data, resources, and models required for model training according to different use scenarios, so as to not need to care about the upstream and downstream associated information of model training. The model generated after model training can be registered in the model warehouse for continuous use of model training or model inference.

[0052] In some embodiments, the method further includes:

[0053] In response to an inference service request, the trained model is published, and the running of the trained model is monitored to dynamically adjust the running configuration of the trained model.

[0054] The running information of the trained model is fed back to use the running information as a data source for optimization of the trained model.

[0055] The inference service module can execute this step, mainly to complete the publishing of the model, control the calling of the model interface, and feed back the calling data of the model interface to the data center for use in model optimization.

[0056] As a whole, a closed loop is formed from data to model to application, and enhancement can be made according to bottlenecks of different modules, such as increasing resources if the resource scheduling module is insufficient, processing data in the data platform if the data required by model training is insufficient, processing models required by inference services in the model warehouse, and the like. Each module is independent and dependent, and each module is connected according to specifications, thereby meeting model production of different use scenarios, improving model development efficiency, and providing a data source for promoting continuous improvement of models.

[0057] In some embodiments, referring to Figure 2 , Figure 2 is a flowchart of steps S201-S202 provided by the embodiments of the present application, the training data is accessed from a data source, and the training data is processed to obtain at least one data set, which can be implemented by steps S201-S202, which will be described in conjunction with each step.

[0058] In step S201, a total data set is newly created, a target version is newly created in the total data set, and the training data is uploaded to the target version. If the training data is greater than or equal to a preset file size, the training data is uploaded in a sharded asynchronous manner. If the training data is less than the preset file size, the training data is uploaded in a synchronous manner. The training data is a compressed file.

[0059] In step S202, the training data is decompressed and previewed, the decompressed training data is labeled and audited, and when the audit is passed, the target version in the total data set is published.

[0060] Here, before starting data processing, a total data set is first newly created. In the total data set, a target version is newly created. Such a structure design is helpful for version management and control of data.

[0061] The training data accessed from the data source is uploaded to the target version in the total data set. The uploading process also considers the file size:

[0062] If the size of the training data is greater than or equal to a preset file size, the training data is uploaded in a sharded asynchronous manner. This manner divides the file into multiple parts and uploads the parts in parallel, which is helpful for improving the uploading speed of large files.

[0063] If the size of the training data is less than the preset file size, the training data is directly uploaded in a synchronous manner. The synchronous uploading manner is relatively simple, but is more suitable for small files because the entire file is uploaded at one time.

[0064] The uploaded compressed training data is decompressed and previewed. This step may be to preliminarily view the content, format and structure of the data.

[0065] After decompression and preview, the data is labeled and audited. Labeling may be to mark or classify certain parts of the data, while auditing is to check the quality and accuracy of the data.

[0066] When the audit is passed, it means that the data is usable and accurate, and at this time the target version in the total data set is released. The released version can be used by other systems or services for subsequent data training and model building.

[0067] The above method focuses on the whole process from uploading to processing and then to releasing the data, and considers the selection of uploading method, data preview, labeling, auditing and other links, to ensure that the final released data set is accurate and reliable.

[0068] As an example, see the following steps:

[0069] 1, a new data set for dialogue model. 2, a new version is created on this data set. 3, upload training data on this version, you can choose a single compressed package file or multiple files on the ring, if the uploaded training data is large, you can compress it into a compressed package for uploading. 4, if it is a large file upload, you can use the sharding method to upload asynchronously, you can interrupt the continuous transmission here, if it is a small file (less than 1M), directly upload synchronously. 5, after uploading successfully, decompress the compressed package and store it in the object storage Minio, you can also upload the decompressed data to cloud storage such as Huawei cloud storage or Ali cloud storage, etc., at the same time, you can preview all the file contents under this data set, if it is a text file, you can view the content of the text by clicking a record. You can also view pictures, audio data here. 6, if you find that the data is abnormal during preview, you can delete it. 7, after the training data is processed, if it is already labeled data, it can be directly used for training task. If it is a raw file, you need to label it on this platform and establish a labeling task. 8, the labeling task is assigned to different labeling personnel for synchronous labeling, and the labeling module can label text, picture and audio data. 9, after the labeling task is processed, it will be sent to the audit personnel for audit. 10, there are several ways of audit, such as sampling audit, overall audit and single audit. Sampling audit is to sample according to a certain proportion, if all are normal, this labeling task is passed, otherwise only one is unqualified and is rejected, and is relabeled. Overall audit is to audit all the labeled data, if all are qualified, it is passed, if one is unqualified, it is rejected and relabeled. Single audit is to audit only one, without concentrating a large amount of labeled data for audit. 11, after the audit is passed, the data set can be published for training task. 12, if the data set is found to be insufficient for the later period, the data can be increased to this version of the data set or a new version can be created on the data set to manage new data. 13, at the same time, if there are multiple data sets of the same type, they can also be combined into a new data set of a version. 14, the data in the data set can be encrypted or not encrypted for export for external system use. 15, integrate the data called by inference service, classify storage, preview, cleaning, download and other operations to provide data source for later optimization model. This is different from other systems in that it centralizes various processing operations of data layer, and connects training task and inference service.

[0070] In some embodiments, the method further comprises:

[0071] in response to a data increase request, supplementing the newly added data in the target version;

[0072] or, in response to a version creation request, creating a new version in the total data set and uploading new version training data in the new version;

[0073] Or, merge the total data set or the target version of the same type.

[0074] Here, when receiving a data increase request, the method does not directly modify the original target version, but supplements the newly added data in the target version. Such design ensures the consistency and integrity of data version, while also supporting dynamic data increase.

[0075] When receiving a version new request, a new version will be created in the total data set, and the training data of the new version will be uploaded in the new version. This is the support for new data or new version management, which reflects the flexibility of the system in data version management.

[0076] The method also supports merging the total data set or the target version of the same type. The merging operation can be based on data similarity, version continuity or other merging strategies, which can effectively integrate resources, reduce redundancy and provide a more comprehensive data view.

[0077] The above methods not only enrich the data processing methods, but also meet the diversified data processing needs, including data increase, version new and data set merging, etc. These methods together constitute a flexible and perfect data processing flow, which can adapt to various data processing and analysis tasks in different scenarios.

[0078] In some embodiments, the selecting a target model matching the access request from the model warehouse comprises:

[0079] Based on the model type information and the model accuracy information, a target model is selected from the model warehouse. If there is no matching target model in the model warehouse, model recommendation information is determined based on the model type information and the model accuracy information, and the target model is obtained based on the model recommendation information.

[0080] Here, when a model needs to be selected, the first consideration is the type of the model (such as classification model, regression model, etc.) and the accuracy of the model. These two factors are usually the key indicators to determine whether a model is suitable for a specific task. The system will first filter the models from the model warehouse based on these two information. The system will search the model warehouse to see if there is a model matching the specified model type and accuracy. If a matching model is found, this model will be selected as the target model.

[0081] If no model matching the request is found in the model repository, the system does not simply return an error or fail. Instead, it generates a model recommendation based on the requested model type and accuracy information. This recommendation may contain suggestions on which model is closest to the request, or which model can be extended or adjusted to meet the request.

[0082] Based on the generated model recommendation, the system may take further actions to obtain or create the target model. For example, the system may suggest that the user download and fine-tune an existing model, or use a pre-trained model as a basis to create a new model.

[0083] The above approach embodies the intelligence and user-friendliness of the system, which not only selects models from the repository based on requests, but also provides valuable suggestions or recommendations to users in the absence of matching models, helping users better complete tasks.

[0084] In some embodiments, the determining the target resource based on the resource specification information comprises:

[0085] Creating a target resource, and assigning the target resource at least one of the following resources based on the resource specification information: algorithm image, algorithm parameters, CPU core number, memory number, GPU model, card number, storage.

[0086] Here, first, based on the provided resource specification information, the system will start creating a target resource. This target resource is a virtual or physical resource needed to meet a specific task or request.

[0087] In the process of creating the target resource, the system will allocate the required resources to the target resource according to the provided resource specification information. These resources include but are not limited to:

[0088] Algorithm image: This is a software image of the algorithm required for machine learning or other computing tasks, which contains all the code and dependencies required to run the algorithm.

[0089] Algorithm parameters: These are configuration parameters required when running the algorithm, which can affect the behavior and performance of the algorithm.

[0090] CPU core number: This indicates the number of central processor cores allocated to the target resource, which determines the ability to handle parallel tasks.

[0091] Memory number: This is the memory capacity allocated to the target resource, which determines the amount of data the resource can handle at the same time.

[0092] GPU model, number of cards: This indicates the model and number of graphics processors allocated to the target resource. Since GPUs are very efficient in handling large-scale parallel computing tasks such as deep learning, this information is crucial for compute-intensive tasks.

[0093] Storage: This is the storage capacity allocated to the target resource, which determines the total amount of data that the resource can store.

[0094] Through this process, the system can create and configure the target resource according to the specific needs of the user, ensuring that the target resource can achieve the best performance and effect when executing specific tasks.

[0095] The method further comprises:

[0096] Monitoring resources during the use of the target resource, dynamically allocating resources in the target resource;

[0097] Or, monitoring resources during the use of the target resource, and sending abnormal resource information if there is an abnormality in the resources in the target resource.

[0098] Here, during the use of the target resource, the system will monitor the resources. This means that the system will continuously check the status of the target resource, including the usage of each resource, such as CPU utilization, memory occupation, GPU usage, storage space, etc.

[0099] Based on the results of resource monitoring, the system can dynamically allocate resources. This means that if a resource appears to be bottlenecked or excessive, the system can automatically adjust the allocation of resources to ensure that the target resource can run efficiently. For example, if the CPU usage is too high, the system may dynamically increase the number of CPU cores or optimize the parallelism of tasks. In actual application, the allocation of resources can also be manually modified based on the monitoring results.

[0100] It should be noted that the dynamic allocation here can be understood as adjusting the application of resources according to the monitoring situation.

[0101] During resource monitoring, if it is detected that there is an abnormality in the resources in the target resource, the system will identify and record it. Abnormalities may include resource over-limit, resource damage, resource access failure, etc.

[0102] When detecting resource abnormalities, the system will send abnormal resource information in a timely manner. This may be through system logs, warning notifications, or API calls to notify relevant parties in order to quickly intervene and repair.

[0103] Through these measures, the method not only ensures the efficient operation of target resources during use, but also timely handles and notifies resource exceptions, ensuring the stability and reliability of the system. This dynamic and exception handling approach makes resource management more intelligent and flexible, better adapting to changing tasks and environments.

[0104] In some implementations, the method further includes:

[0105] In response to the trained model meeting the preset condition, the trained model is added to the model warehouse.

[0106] Here, when the trained model meets certain preset conditions, the model will be considered for addition to the model warehouse.

[0107] The "preset condition" here can involve multiple aspects:

[0108] Performance indicators: For example, the accuracy, recall rate, F1 score, etc. of the model on the validation set meet certain preset standards.

[0109] Resource usage: The model meets specific resource constraints during training or inference, such as CPU, GPU, or memory usage.

[0110] Model stability: After multiple training and validation, the performance of the model is stable and there is no large fluctuation.

[0111] Other business rules: Depending on the actual application scenario, there may be other specific conditions, such as model size, inference speed, etc.

[0112] When the trained model meets these preset conditions, it will be added to the "model warehouse". The model warehouse is a place to store and manage multiple models, which helps to organize, version control and deploy models. Adding a model to the model warehouse means that the model has been considered valuable and may be used for actual deployment, prediction or other tasks in the future.

[0113] In summary, this is a quality control and management process that ensures only models that meet certain standards are included in the model warehouse, providing high-quality model selection for subsequent applications.

[0114] In summary, the embodiments of the present application have the following beneficial effects:

[0115] By decoupling each stage of model production, each stage can complete its own function and provide module capabilities to the outside. This can ensure independent operation of each stage and enhance the operation efficiency of the entire system, without causing changes in the entire system due to changes in a certain stage. This has good support for system scalability, robustness, reusability, and portability. In addition to being able to self-cycle, the model can be optimized and upgraded step by step, and the existing capabilities of third parties can be integrated. Not only can the achievements of third parties be used, but more suitable models can also be produced based on the capabilities of third parties according to different business scenarios.

[0116] Based on the same inventive concept, the model production apparatus corresponding to the model production method in the first embodiment is also provided in the embodiments of the present application. Since the principle of the apparatus in the embodiments of the present application for solving the problem is similar to the above-mentioned model production method, the implementation of the apparatus can be referred to the implementation of the method, and the repeated parts will not be described again.

[0117] As shown in Figure 3 , Figure 3 is a structural schematic diagram of a model production apparatus 300 provided by the embodiments of the present application. The model production apparatus 300 comprises:

[0118] a data platform module 301 configured to access training data from a data source in response to a data access request, and process the training data to obtain a target data set;

[0119] a model warehouse module 302 configured to select a target model matching the access request from a model warehouse in response to a model access request, wherein the access request carries model type information and model accuracy information;

[0120] a resource scheduling module 303 configured to determine a target resource through resource specification information in response to a resource scheduling request, wherein the resource specification information is determined based on the target data set and the target model;

[0121] a training task module 304 configured to execute a training task based on the target resource in response to a model training request, wherein the training task is created based on the target data set and the target model.

[0122] Those skilled in the art should understand that Figure 3 the implementation functions of each unit in the model production apparatus 300 shown in the above can be understood with reference to the related description of the foregoing model production method. Figure 3 The functions of each unit in the model production apparatus 300 shown in the above can be implemented by a program running on a processor, or by a specific logic circuit.

[0123] In a possible implementation, the apparatus further comprises:

[0124] The inference service module 305 is configured to publish the trained model in response to an inference service request, and monitor the running of the trained model to dynamically adjust the running configuration of the trained model.

[0125] The running information of the trained model is fed back to serve as a data source for optimizing the trained model.

[0126] In a possible implementation, the data middle platform module 301 accesses training data from a data source and processes the training data to obtain at least one data set, including:

[0127] A total data set is created, a target version is created in the total data set, the training data is uploaded to the target version, if the training data is greater than or equal to a preset file size, the training data is uploaded in a sharded asynchronous manner, if the training data is less than the preset file size, the training data is uploaded in a synchronous manner, and the training data is a compressed file.

[0128] The training data is decompressed and previewed, the decompressed training data is labeled and audited, and when the audit is passed, the target version in the total data set is published.

[0129] In a possible implementation, the data middle platform module 301 further includes:

[0130] In response to a data increase request, new data is supplemented in the target version.

[0131] Alternatively, in response to a version creation request, a new version is created in the total data set, and new version training data is uploaded in the new version.

[0132] Alternatively, the total data set or the target version of the same type is merged.

[0133] In a possible implementation, the model warehouse module 302 selects a target model matched with the access request from a model warehouse, including:

[0134] The target model is selected from the model warehouse based on the model type information and the model accuracy information, if the model warehouse does not have the matched target model, model recommendation information is determined based on the model type information and the model accuracy information, and the target model is obtained based on the model recommendation information.

[0135] In a possible implementation, the resource scheduling module 303 determines a target resource based on resource specification information, including:

[0136] Create a target resource, and allocate at least one of the following resources to the target resource based on the resource specification information: algorithm image, algorithm parameters, number of CPU cores, number of memory, GPU model, number of cards, and storage;

[0137] Resource scheduling module 303 also includes:

[0138] Resource monitoring is performed during the use of the target resource, and resources in the target resource are dynamically allocated;

[0139] Alternatively, resource monitoring can be performed during the use of the target resource, and if any abnormality is found in the target resource, abnormal resource information can be sent.

[0140] In one possible implementation, the inference service module 305 further includes:

[0141] If the trained model meets the preset conditions, the trained model is added to the model repository.

[0142] The aforementioned model production device decouples the various stages of model production, allowing each stage to complete its own function and provide modular capabilities. This ensures the independent operation of each stage while simultaneously enhancing the overall system's operational efficiency, preventing changes in any single stage from affecting the entire system. It provides excellent support for the system's scalability, robustness, reusability, and portability. Furthermore, in addition to self-circulation, it ensures the gradual optimization and upgrading of models and can integrate existing third-party capabilities. This allows users to not only utilize third-party achievements but also to produce models more suited to different business scenarios based on those capabilities.

[0143] like Figure 4 As shown, Figure 4 This is a schematic diagram of the composition structure of the electronic device 400 provided in the embodiments of this application. The electronic device 400 includes:

[0144] The device 400 includes a processor 401, a storage medium 402, and a bus 403. The storage medium 402 stores machine-readable instructions that can be executed by the processor 401. When the electronic device 400 is running, the processor 401 communicates with the storage medium 402 via the bus 403. The processor 401 executes the machine-readable instructions to perform the steps of the model production method described in the embodiments of this application.

[0145] In practical applications, the various components in the electronic device 400 are coupled together via a bus 403. It is understood that the bus 403 is used to achieve communication between these components. In addition to a data bus, the bus 403 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in... Figure 4 The general designated all buses as Bus 403.

[0146] The electronic device described above decouples each stage of model production, so that each stage completes its own function and provides module capabilities externally. This ensures independent operation of each stage and simultaneously enhances the operation efficiency of the entire system, so that changes in a certain stage do not cause changes in the entire system. The system has good support for scalability, robustness, reusability, and portability. In addition to being able to self-cycle, the model can be gradually optimized and upgraded, and the existing capabilities of third parties can be integrated. Not only can the achievements of third parties be used, but more suitable models can also be produced based on the capabilities of third parties according to different business scenarios.

[0147] The embodiment of the present application also provides a computer readable storage medium, the storage medium stores executable instructions, when the executable instructions are executed by at least one processor 401, the model production method described in the embodiment of the present application is realized.

[0148] In some embodiments, the storage medium can be a ferromagnetic random access memory (FRAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM), etc. It can also be various devices including one or any combination of the above memories.

[0149] In some embodiments, the executable instructions can be in the form of programs, software, software modules, scripts or codes, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and can be deployed in any form, including as a standalone program or as a module, component, subroutine or other unit suitable for use in a computing environment.

[0150] As an example, executable instructions can correspond to a file in a file system, can be stored in a part of a file that is used to store other programs or data (e.g., one or more scripts stored in a markup language despatch, HTML (HyperText Markup Language) document), can be stored as a single file dedicated to the program in question, or can be stored in multiple coordinated files (e.g., files that store one or more modules, subprograms, or portions of code).

[0151] As an example, executable instructions can be deployed to execute on one computing device, or on multiple computing devices located at one site, or on multiple computing devices distributed across multiple sites and interconnected through a communication network.

[0152] The above computer-readable storage medium decouples each stage of model production, achieves respective functions of each stage, and provides module capabilities. The independent operation of each stage is ensured, and the operation efficiency of the entire system is enhanced at the same time. The expansibility, robustness, reusability, and portability of the system are well supported. In addition to self-circulation, the gradual optimization and upgrading of the model are ensured, and the existing capabilities of third parties can be integrated. Not only the achievements of third parties can be used, but also more needed models can be produced based on the capabilities of third parties according to different business scenarios.

[0153] In several embodiments provided in the present application, it should be understood that the disclosed method and electronic device can be implemented in other ways. The above-described device embodiments are only illustrative. For example, the division of the units is only a logical function division, and actual implementation can have another division manner. For example, multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed components can be indirect coupling or communication connection through some interfaces, devices or units, and can be electrical, mechanical or other forms.

[0154] The modules described as separate components can or can not be physically separated, and the components shown as modules can or can not be physical units, that is, they can be located in one place or distributed on multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the present embodiment.

[0155] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit.

[0156] The functions described can be implemented in hardware, software, firmware or any combination thereof. If implemented in software, the functions can be stored or transmitted over as one or more instructions or code on a computer-readable medium. Computer-readable media include both computer storage media and communication media including any medium that facilitates transfer of a computer program from one place to another. A storage medium can be any available media that can be accessed by a computer. By way of example, and not limitation, such computer-readable media can comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer. Also, any connection is properly termed a computer-readable medium. For example, if the software is transmitted from a website, server, or other remote source using a coaxial cable, fiber optic cable, or twisted pair, then the coaxial cable, fiber optic cable, or twisted pair are included in the definition of medium. Disk and disc, as used herein, include compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), and Blu-Ray® disc where disks usually reproduce data magnetically, while discs reproduce data optically with lasers. Combinations of the above should also be included within the scope of computer-readable media.

[0157] In the above detailed description, only specific embodiments of the present application have been described. However, it should be apparent that a person of ordinary skill in the art can make changes and modifications to the present application without departing from the spirit and scope thereof. Therefore, the scope of the present application should be determined by the following claims.

Claims

1. A model production method characterized by, The method comprises: accessing training data from a data source in response to a data access request, and processing the training data to obtain a target data set; selecting a target model matching the access request from a model repository in response to a model access request, wherein the access request carries model type information and model accuracy information; determining a target resource through resource specification information in response to a resource scheduling request, wherein the resource specification information is determined based on the target data set and the target model; executing a training task based on the target resource in response to a model training request, wherein the training task is created based on the target data set and the target model; the method further comprises: creating a total data set and a target version in the total data set, uploading the training data to the target version, if the training data is greater than or equal to a preset file size, uploading the training data through a sharded asynchronous manner, if the training data is less than the preset file size, uploading the training data through a synchronous manner, and the training data is a compressed file; decompressing and previewing the training data to label and audit the decompressed training data, and publishing the target version in the total data set after the audit is passed; the method further comprises: in response to a data increase request, supplementing new data in the target version; or, in response to a version creation request, creating a new version in the total data set and uploading new version training data in the new version; or, merging the total data set or the target version of the same type; the method further comprises: based on the model type information and the model accuracy information, selecting a target model from the model repository, if the model repository does not exist, determining model recommendation information based on the model type information and the model accuracy information, and obtaining the target model based on the model recommendation information; the method further comprises: in response to an inference service request, publishing a trained model and monitoring the running of the trained model to dynamically adjust the running configuration of the trained model; and, returning the running information of the trained model to use the running information as a data source for optimizing the trained model; the method further comprises: creating a target resource, and allocating the following resources to the target resource based on the resource specification information: algorithm image, algorithm parameter, CPU core number, memory number, GPU model, card number, and storage; the method further comprises: monitoring resources during the use of the target resource, and dynamically allocating resources in the target resource; or, monitoring resources during the use of the target resource, and sending abnormal resource information if there is an abnormality in the resources in the target resource; the method further comprises: In response to the trained model meeting a preset condition, the trained model is added to the model warehouse.

2. A model production apparatus characterized by comprising: The device comprises: The data hub module is configured to access training data from a data source and process the training data to obtain a target data set in response to a data access request; the accessing of the training data from the data source and the processing of the training data to obtain at least one data set comprises: creating a total data set, creating a target version in the total data set, uploading the training data to the target version, uploading the training data by a sharded asynchronous manner if the training data is greater than or equal to a preset file size, uploading the training data by a synchronous manner if the training data is less than the preset file size, the training data being a compressed file; performing decompression preview processing on the training data to label and audit the decompressed training data, publishing the target version in the total data set after the audit is passed; the data hub module is further configured to supplement new data in the target version in response to a data increase request, create a new version in the total data set in response to a version creation request, and upload new version training data in the new version, or merge the total data set or the target version of the same type; The model warehouse module is configured to select a target model matched with the access request from the model warehouse in response to a model access request, wherein the access request carries model type information and model accuracy information; the selecting of the target model from the model warehouse in response to the access request comprises: selecting the target model from the model warehouse based on the model type information and the model accuracy information, determining model recommendation information based on the model type information and the model accuracy information if the model warehouse does not have the target model matched with the access request, and obtaining the target model based on the model recommendation information; The resource scheduling module is configured to determine a target resource based on resource specification information in response to a resource scheduling request, wherein the resource specification information is determined based on the target data set and the target model; The training task module is configured to execute a training task based on the target resource in response to a model training request, wherein the training task is created based on the target data set and the target model; The data hub module is further configured to publish a trained model in response to an inference service request, monitor running of the trained model to dynamically adjust running configuration of the trained model, and return running information of the trained model to use the running information as a data source for optimizing the trained model; The resource scheduling module is specifically configured to create a target resource, and allocate at least one of the following resources to the target resource based on the resource specification information: an algorithm image, an algorithm parameter, a CPU core number, a memory number, a GPU model, a card number, and storage. The resource scheduling module is further configured to perform resource monitoring during use of the target resource, perform dynamic allocation of resources in the target resource, or perform resource monitoring during use of the target resource and send abnormal resource information if there is an abnormality in the resources in the target resource. The model warehouse module is further configured to add the trained model to the model warehouse in response to the trained model satisfying a preset condition.

3. An electronic device, comprising: The model production method comprises the following steps: A processor, a storage medium and a bus, wherein the storage medium stores machine readable instructions executable by the processor, the processor and the storage medium communicate through the bus when the electronic device is running, and the processor executes the machine readable instructions to perform the model production method of claim 1.

Images