Model generation method and device, medium and equipment

By customizing the terminal device model, model tasks and model constraints to generate a personalized model, the problem of insufficient adaptability of existing NAS algorithms in terminal devices is solved, and efficient resource utilization and automated personalized model generation is realized.

CN118940801APending Publication Date: 2024-11-12SHENZHEN HEYTAP TECHNOLOGY CO LTD
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
CN202310525339.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-05-10
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

The existing neural architecture search (NAS) algorithms are less personalized when generating machine learning models, and it is difficult to adapt to hardware differences between different terminal devices, resulting in insufficient resource utilization and performance impacts during model deployment.

Method used

Through custom operations, determine the terminal device model, model tasks and model constraints, generate a personalized target model, adapt to specific terminal device resources, and achieve efficient matching between the model and the device.

Benefits of technology

It improves the adaptability of the model and terminal equipment, realizes the rational use of resources, and improves the degree of automation and personalization in the model generation process.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118940801A_ABST
    Figure CN118940801A_ABST
Patent Text Reader

Abstract

The invention provides a model generation method, a model generation device, a computer readable storage medium and electronic equipment, and relates to the technical field of machine learning, and the method can consider hardware differences existing between equipment, provides a user-defined function, and can improve the user experience for a user. According to the method, the terminal equipment model, the model task and the model constraint condition can be selected through the user-defined operation, the personalized target model can be generated according to the terminal equipment model, the model task and the model constraint condition, the target model and the terminal equipment are higher in adaptability, and reasonable utilization of terminal equipment resources is easier to achieve.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the technical field of machine learning, and in particular to a model generation method, a model generation device, a computer-readable storage medium, and an electronic device. Background Art

[0002] Automated Machine Learning (Auto-ML) refers to automating the steps of data preprocessing, feature selection, algorithm selection in machine learning, as well as the steps of neural network architecture design, hyperparameter optimization, and neural network model training in deep learning, so that the desired results can be obtained without human intervention.

[0003] Neural architecture search (NAS) algorithms can be used to automatically discover ideal ML models for specific tasks. Generally speaking, the use of neural architecture search (NAS) algorithms combined with data sets and model tasks can automatically generate suitable ML models. However, this generation method has the problem of low personalization.

[0004] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present application, and therefore may include information that does not constitute relevant technology known to ordinary technicians in the field. Summary of the invention

[0005] The purpose of the present application is to provide a model generation method, a model generation device, a computer-readable storage medium and an electronic device, which can take into account the hardware differences between devices and provide customized functions. For users, terminal device models, model tasks, and model constraints can be selected through customized operations. The present application can generate personalized target models for terminal device models, model tasks, and model constraints. The target model has higher adaptability to the terminal device and makes it easier to achieve rational use of terminal device resources.

[0006] Other features and advantages of the present application will become apparent from the following detailed description, or may be learned in part by the practice of the present application.

[0007] According to one aspect of the present application, a model generation method is provided, the method comprising:

[0008] In response to the custom operation, determining a custom terminal device model, model task, and model constraint condition;

[0009] Generate a target model for deployment on the terminal device based on the terminal device model, model tasks, and model constraints.

[0010] According to one aspect of the present application, a model generation device is provided, the device comprising:

[0011] A custom information determination unit, used to determine a custom terminal device model, model task, and model constraint condition in response to a custom operation;

[0012] The model generation unit is used to generate a target model for deployment on the terminal device according to the terminal device model, model tasks, and model constraints.

[0013] According to one aspect of the present application, a computer program product or a computer program is provided, the computer program product or the computer program comprising computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the methods provided in the above-mentioned various optional implementations.

[0014] According to one aspect of the present application, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, any one of the above methods is implemented.

[0015] According to one aspect of the present application, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to perform any one of the above methods by executing the executable instructions.

[0016] The exemplary embodiments of the present application may have some or all of the following beneficial effects:

[0017] In the model generation method provided in an example implementation of the present application, hardware differences between devices can be taken into account to provide a custom function. For users, terminal device models, model tasks, and model constraints can be selected through custom operations. The present application can generate a personalized target model for the terminal device model, model tasks, and model constraints. The target model is more adaptable to the terminal device and is more likely to achieve reasonable utilization of terminal device resources. In addition, since the present application can automatically generate a personalized target model for the terminal device, the present application simultaneously realizes automation and personalization in the model generation process.

[0018] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present application, and together with the specification are used to explain the principles of the present application. Obviously, the drawings described below are only some embodiments of the present application, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0020] Figure 1 A flow chart of a model generation method according to an embodiment of the present application is schematically shown;

[0021] Figure 2 A schematic diagram of an introduction interface of a data access function according to an embodiment of the present application is schematically shown;

[0022] Figure 3 Schematically shows an interface diagram including a statistical analysis function according to an embodiment of the present application;

[0023] Figure 4 A schematic diagram of an interface including a model task according to an embodiment of the present application is schematically shown;

[0024] Figure 5 A flow chart of a model generation method according to another embodiment of the present application is schematically shown;

[0025] Figure 6 The structure diagram of a model generation device according to an embodiment of the present application is schematically shown;

[0026] Figure 7 The structure diagram of a computer system suitable for implementing an electronic device of an embodiment of the present application is schematically shown. DETAILED DESCRIPTION

[0027] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as being limited to the examples set forth herein; on the contrary, these embodiments are provided so that the present application will be more comprehensive and complete, and the concept of the example embodiments will be fully conveyed to those skilled in the art. The described features, structures, or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present application. However, those skilled in the art will appreciate that the technical solutions of the present application may be practiced while omitting one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known technical solutions are not shown or described in detail to avoid obscuring various aspects of the present application.

[0028] See also Figure 1 , Figure 1 The following schematically shows a flow chart of a model generation method according to an embodiment of the present application. Figure 1 As shown, the method includes the following steps.

[0029] Step S110: In response to the customization operation, determine the customized terminal device model, model task, and model constraint condition.

[0030] Step S120: Generate a target model for deployment on the terminal device according to the terminal device model, model tasks, and model constraints.

[0031] Implementation Figure 1 The method shown can take into account the hardware differences between devices and provide custom functions. For users, they can select terminal device models, model tasks, and model constraints through custom operations. The present application can generate personalized target models for terminal device models, model tasks, and model constraints. The target model is more compatible with the terminal device and is easier to achieve reasonable use of terminal device resources. In addition, since the present application can automatically generate personalized target models for terminal devices, the present application simultaneously realizes automation and personalization in the model generation process.

[0032] Next, the above steps of this exemplary embodiment are described in more detail.

[0033] In step S110, in response to the customization operation, a customized terminal device model, model task, and model constraint condition are determined.

[0034] In order to solve some classification, detection, recognition and other problems, we can design corresponding deep learning models. By training these models with specific data sets, we can enable deep learning models to have the ability to solve the above problems. Generally speaking, we can deploy deep learning models on designated terminal devices so that they can play their corresponding functions on designated terminal devices.

[0035] However, deep learning models are usually large in size and require more device resources. They are also computationally intensive and memory intensive. Deep learning models can be successfully deployed and applied to terminal devices with rich hardware resources (such as servers, etc.). However, for terminal devices with limited hardware resources (such as mobile phones, tablets, smart watches, smart bracelets, etc.), on the one hand, it is difficult to deploy ordinary deep learning models, and on the other hand, deploying deep learning models is likely to affect device performance.

[0036] In related technologies, deep learning models are usually designed for a class of terminal devices based on a proxy method through neural architecture search. However, this method also has limitations. Because there are various models in a class of devices, the configurations of different terminal devices (such as CPU parallelism, cache size, memory bandwidth, amount of information accessed by the memory per unit time, etc.) are also different. If the deep learning model is to be adapted to the terminal device as much as possible, so that the deep learning model can obtain better accuracy and higher hardware efficiency on the target hardware device, the model cannot be generated in the way of related technologies.

[0037] The present application contemplates that the terminal device model, model task, and model constraint conditions can all be used as model generation conditions; wherein the terminal device model can be represented by characters, numbers, symbols, etc., and different terminal device models correspond to different hardware configurations; the model task can be used to identify the design purpose of the model, for example, the model task is object classification, object detection, semantic segmentation, instance segmentation, object tracking, etc.; the model constraint conditions can be used to limit the effect of the model, for example, the model constraint conditions include at least one of the conditions such as inference latency, inference power consumption, model size, running memory consumption, and model computational amount (FLOPs).

[0038] In step S120, a target model for deployment on a terminal device is generated according to the terminal device model, model tasks, and model constraints.

[0039] Specifically, the target model deployed on the terminal device can be any deep learning model that meets the requirements, such as ResNet, DenseNet, MobileNet, ShuffleNet, etc., which is not limited in the embodiments of the present application.

[0040] As an optional embodiment, a target model for deployment on a terminal device is generated according to the terminal device model, model task, and model constraint conditions, including: in response to a setting operation, determining a set training reference condition; wherein the model training reference condition includes at least one of a neural network block, a model form, a model input size, and a model output size; and a target model for deployment on a terminal device is generated according to the terminal device model, model task, model constraint conditions, and model training reference conditions. In this way, a target model that better meets user needs and adapts to terminal devices can be generated based on conditions in more dimensions.

[0041] Specifically, in addition to the terminal device model, model task, and model constraint conditions, a visual interface can also be provided for users to set at least one of the following conditions: neural network block, model form, model input size, and model output size, and then use these conditions as model generation conditions. Among them, a neural network block can describe a single layer, a component composed of multiple layers, or the entire model itself. The advantage of using a neural network block for abstraction is that some neural network blocks can be combined into larger components. This process is usually recursive. In the neural network block, the corresponding architecture search parameters and the combination principles between various operations are included, which can be applied in the model search process. Users can implement addition or deletion operations for neural network blocks. Model form refers to the style of the model (such as UNet type, CNN classification type, Transformer type, etc.); model form, model input size, and model output size can be used to jointly characterize the search space of the model, among which the model input size can be used to define the data scale input to the model, and the model output size can be used to define the data scale of the model output.

[0042] In addition, optionally, a mapping table may be included in the visualization interface provided to the user, which can be used to characterize the mapping relationship between model tasks and model forms. The user can directly select the corresponding model form according to the model task, or define the model form by himself as described above, which is not limited to the embodiments of the present application.

[0043] As an optional embodiment, a target model for deployment on a terminal device is generated according to the terminal device model, model tasks, and model constraints, including: obtaining a sample data set, and generating a first model according to the sample data set, model tasks, and model constraints; and saving the first model as a target model for deployment on the terminal device. In this way, not only can model generation based on sample data sets, model tasks, and model constraints be achieved, but the model can also be saved as a target model suitable for terminal device reasoning, which can save manual design costs and improve the generation efficiency of personalized models.

[0044] Specifically, the sample data set contains any number of sample data, and the sample data can be understood as labeled data, that is, each sample data has a corresponding label so that the model can be trained accordingly.

[0045] As an optional embodiment, obtaining a sample data set includes: displaying a data source type in response to a data source creation request; determining a target data source type in response to a selection operation on the data source type; and importing the sample data set into a model training platform based on the target data source type. This makes it easier for users to customize data source types, and then import the sample data set into the model training platform based on the defined data source type, which can improve interactivity with users.

[0046] Specifically, data source types can include: Hive, MySQL, PostgreSQL, HDFS, MongoDB, etc. After the user logs in to the model training platform, an introduction interface about the data access function can be displayed to the user. Please refer to Figure 2 , Figure 2 The following schematic diagram shows an interface diagram of the data access function according to an embodiment of the present application. After the user logs in to the model training platform, he can browse to Figure 2 The introduction of data access function shown in the figure, then, based on Figure 2 The content shown can realize gradual data access so as to implement model training in the model training platform.

[0047] Among them, the model training platform can be used to generate corresponding models for terminal devices in a targeted manner and push the models to the terminal devices. After the terminal devices deploy the pushed models, the corresponding detection, classification and other functions can be realized through the models.

[0048] It should be noted that Figure 2 The diagram is only a schematic diagram. In actual application, the displayed introduction content and interface design may be different from Figure 2 , the embodiments of the present application are not limited thereto.

[0049] As an optional embodiment, it also includes: performing statistical analysis on the sample data set through the model training platform to obtain statistical analysis results; in response to a viewing operation on the statistical analysis results, displaying the statistical analysis results; in response to an automatic balancing request for the statistical analysis results, performing a data enhancement operation to balance the amount of data corresponding to each label in the sample data set. In this way, an automated data analysis function can be provided to the user, and the results of the data analysis can be displayed to the user so that the user can decide whether to start the data enhancement function, which can improve the interactivity with the user.

[0050] Specifically, the model training platform can also implement statistical analysis functions for sample data sets. After the user logs in to the model training platform, an interface containing statistical analysis functions can also be displayed to the user, such as Figure 3 As shown, Figure 3 The schematic diagram of an interface including a statistical analysis function according to an embodiment of the present application is schematically shown. The interface can display a variety of visual content, such as experiments, components, historical cycles, task records, favorites, etc. Among them, for experiments, historical cycles, task records, and favorites, the specific displayed content can be determined according to actual needs, and the embodiment of the present application does not limit it.

[0051] The component column includes a variety of specific components that can be provided to users for data management. Regarding data reading and writing, the components included are at least: reading Hive, reading HDFS, reading MySQL, reading PostgreSQL, reading MongoDB, and reading OCS. It can be understood that the data types that these components can read include at least Hive, MySQL, PostgreSQL, HDFS, MongoDB, and OCS. That is, the model training platform supports the multiple data types described above.

[0052] Regarding data preprocessing, the components included are at least: processing outliers, processing missing values, customization, and data connection (JION). Among them, the component for processing outliers is used to filter out abnormal data that does not conform to the rules in the sample data set, the component for processing missing values ​​is used to filter out data with missing values ​​or missing labels in the sample data set, the customized component is used to provide customized processing functions for the data, and the data connection (JION) component is used to provide connection functions between data tables, which can specifically include left connection (LEFT JION) and right connection (RIGHT JION) and other functions.

[0053] Regarding statistical analysis, the components included are at least: deviation analysis, histogram, scatter plot, and table statistics, among which the deviation analysis component is used to perform deviation analysis on data in a sample data set, the histogram component is used to generate a histogram for a sample data set, which can also be understood as representing the sample data set as a histogram, the scatter plot component is used to generate a scatter plot for a sample data set, which can also be understood as representing the sample data set as a scatter plot, and the table statistics component is used to count the sample data in the sample data set in a data table. The basis for the statistics can be any one or more conditions such as time, label, etc., which is not limited to the embodiments of the present application.

[0054] It should be noted that Figure 3 The diagram is only a schematic diagram. In actual application, Figure 3 The interface may include other contents in addition to the statistical analysis function, which is not limited in the embodiments of the present application.

[0055] Furthermore, after statistical analysis is performed on the sample data set through the model training platform and the statistical analysis results are obtained, the statistical analysis results can be displayed in response to the user's viewing operation on the statistical analysis results; wherein the statistical analysis results can be expressed in any form such as text, graph, table, etc., and the statistical analysis results can include statistical results based on at least one of data category, quantity, and type distribution.

[0056] As an optional embodiment, statistical analysis is performed on the sample data set through the model training platform to obtain statistical analysis results, including: performing at least one of deviation analysis, histogram analysis, scatter plot analysis, and table statistics on the sample data set through the model training platform to obtain statistical analysis results. This can provide users with more diverse data statistics functions, so that users can perform subsequent data processing and other operations.

[0057] Specifically, deviation analysis can be used to analyze data deviations between sample data in a sample data set; histogram analysis can be used to concentrate sample data in a sample data set in a histogram, and intuitively present the differences between data to users through the representation of the histogram; scatter plot analysis can be used to concentrate sample data in a sample data set in a scatter plot, and users can intuitively understand the distribution of sample data in the sample data set based on the scatter plot; table statistics can be used to concentrate sample data in a sample data set and summarize them in a data table.

[0058] As an optional embodiment, a data enhancement operation is performed to balance the amount of data corresponding to each label in the sample data set, including: if the sample data set is an audio data set, then the amount of data corresponding to each label in the sample data set is balanced by at least one processing means of noise enhancement, audio splicing, time shift enhancement, pitch change, speed adjustment, volume adjustment, mixed background sound, white noise supplement, mobile audio, and stretched audio; if the sample data set is an image data set, then the amount of data corresponding to each label in the sample data set is balanced by at least one processing means of horizontal flipping, vertical flipping, rotation, scaling, cropping, translation, Gaussian noise, generative adversarial network, and screenshot; if the sample data set is a text data set, then the amount of data corresponding to each label in the sample data set is balanced by at least one processing means of syntax tree structure replacement, chapter interception, sequence generation, generative adversarial network, and pre-trained language model. In this way, the amount of data of each label in the sample data set can be balanced to avoid the situation where the sample amount of a label is too small, which is conducive to improving the training effect of the model and improving the accuracy of model training.

[0059] Specifically, the sample data set may not be limited to an audio data set, an image data set, or a text data set, but may also be other types of data sets, which are not limited in the embodiments of the present application. Among them, the data enhancement means for the sample data set may be used for labels with a small sample size in the sample data set, and the judgment standard for labels with a small sample size is: the data volume between each two labels is compared, and if the data volume difference is greater than a preset difference, it is determined that a label with a small sample size is detected.

[0060] In addition, noise enhancement is used to increase the noise intensity in the audio data, and the audio data with increased noise intensity is added as new sample data in the sample data set; audio splicing is used to splice at least two audio data in the sample data set as a whole or in part (for example, splicing a segment of audio data A with a segment of audio data B), and the obtained new audio data is added to the sample data set; time shift enhancement is used to increase the time shift intensity in the audio data, and the audio data with increased time shift intensity is added as new sample data in the sample data set; pitch conversion is used to change the pitch of the audio data, and the obtained new audio data is added to the sample data set; speed adjustment is used to change the playback speed of the audio data, and the obtained new audio data is added to the sample data set. The data is supplemented in the sample data set; volume adjustment is used to change the volume of audio data, and the obtained new audio data is supplemented in the sample data set; mixed background sound is used to mix the background sounds between different audio data, and the obtained new audio data is supplemented in the sample data set; white noise supplement is used to supplement white noise to audio data, and the obtained new audio data is supplemented in the sample data set. In addition, in addition to supplementing white noise, other types of noise can also be supplemented in the audio data to enrich the sample data set; moving audio is used to move audio fragments in audio data, and the obtained new audio data is supplemented in the sample data set; stretching audio is used to stretch audio data, and the obtained new audio data is supplemented in the sample data set.

[0061] In addition, the image data in the sample data set can be horizontally flipped, vertically flipped, rotated, scaled, cropped, and translated to obtain new image data, and the obtained new image data can be added to the sample data set. In addition, Gaussian noise can be added to the image data in the sample data set to obtain new image data, and the obtained new image data can be added to the sample data set. And, it is also possible to train a generative adversarial network so that the trained generative adversarial network generates new image data, and the obtained new image data can be added to the sample data set. In addition, the image data in the sample data set can be screenshotted to obtain new image data, and the obtained new image data can be added to the sample data set.

[0062] In addition, syntax tree structure replacement can replace words or characters in the text data in the sample data set based on the syntax tree to obtain new text data, and add the obtained new text data to the sample data set; chapter extraction can be used to extract local data of the text data in the sample data set to obtain new text data, and add the obtained new text data to the sample data set; sequence generation refers to the automatic generation of new text data, and adding the obtained new text data to the sample data set; in addition, it is also possible to train a generative adversarial network so that the trained generative adversarial network generates new text data, and add the obtained new text data to the sample data set; in addition, it is also possible to generate text data for supplementing the sample data set through a pre-trained language model.

[0063] As an optional embodiment, a first model is generated according to a sample data set, a model task, and a model constraint condition, including: determining the hardware information corresponding to the terminal device model from a hardware information library; selecting a model architecture based on a binary gate based on the hardware information and the model task to obtain a plurality of reference models; wherein the binary gate is used to limit the architecture parameters based on which the model architecture is selected each time; training a plurality of reference models based on the sample data set to obtain a plurality of intermediate models; and determining the first model from the plurality of intermediate models based on the model constraint condition. In this way, the model architecture can be selected based on the set binary gate to determine an architecture suitable for participating in the model training, and the architecture is more suitable for a specified terminal device, which can be conducive to more efficient generation of a personalized model for deployment on the terminal device.

[0064] Specifically, the hardware information corresponding to the terminal device model may include: CPU chip, GPU chip / TPU chip, memory, storage, bandwidth, etc. For different terminal device models, the corresponding hardware information is stored in the hardware information library, so there is no need for users to manually enter the configuration. By reading the corresponding hardware information in the hardware information library, the model architecture based on binary gate can be selected based on the hardware information and model tasks to obtain a variety of reference models.

[0065] In addition, the hardware information library can also be used to record the deep learning model operation results corresponding to different hardware information. The results can be used to characterize the advantages and disadvantages of the terminal device corresponding to the hardware information when inferring the deep learning model.

[0066] For the selection of model tasks, an interface including model tasks can be provided to the user, such as Figure 4 As shown, Figure 4The schematic diagram of the interface including model tasks according to an embodiment of the present application is schematically shown. Among them, different model tasks can be displayed for user classification for user visual selection. For example, the model tasks included in binary classification include at least: XGBoost binary classification, random forest binary classification, GBDT binary classification, decision tree binary classification; the model tasks included in multi-classification include at least: XGBoost multi-classification, logistic regression, random forest multi-classification, GBDT multi-classification.

[0067] In this process, only one node (operation, OP) between each neural network layer is activated, and the other OPs are in an inhibited state. A TensorFlow node (operation, OP) is a node in a computational graph that is used to run computational operations performed on tensors. It accepts zero or more tensors as input and produces zero or more tensors as output.

[0068] Among them, the binary gate can be set in the overall model architecture, and the binary gate contains a numerical value (such as 1, 0, 0, ..., 0) corresponding to each architecture parameter (such as α, β, σ, ..., δ), wherein the architecture parameter and the numerical value are in a one-to-one correspondence. Each architecture parameter pair is used for a layer of neural network. In each round of model architecture selection, a new model architecture selection can be achieved by adjusting the binary gate to obtain a reference model. Furthermore, the reference model can be trained in a cloud real machine consistent with the terminal device model to obtain an intermediate model, and the loss function Loss corresponding to the intermediate model is used. CE As well as the values ​​corresponding to various model constraints (inference latency, inference power consumption, model size, running memory consumption, model computing power (FLOPs)), we can calculate

[0069] Based on Loss, an optimal model can be selected from multiple intermediate models as the first model. Used to describe the sparsity of the network structure; λ1, λ2, λ3, λ4, λ % , λ6 are weight values ​​set according to actual conditions.

[0070] Regarding the changes in binary gates, for example, the binary gate (1, 0, 0, ..., 0) in the first round can be adjusted to a binary gate (0, 1, 0, ..., 0), and the model architecture selection for the second round can be performed based on (0, 1, 0, ..., 0). Furthermore, in the third round, the binary gate (0, 1, 0, ..., 0) can be adjusted to a binary gate (0, 0, 1, ..., 0).

[0071] As an optional embodiment, the first model is saved as a target model for deployment on a terminal device, including: performing network compression on the first model to obtain a second model; wherein the parameter amount of the second model is less than the parameter amount of the first model; performing quantitative compression on the second model to obtain a third model; wherein the storage occupancy of the third model is less than the storage occupancy of the second model; saving the third model as a target model for deployment on a terminal device; wherein the target model and the third model correspond to different storage formats. In this way, based on network compression and quantitative compression, the volume of the model that needs to be deployed on the terminal device can be reduced, the performance of the terminal device can be avoided from being affected to improve the operating efficiency of the terminal device, and the model deployment efficiency can be improved.

[0072] Specifically, the method of quantizing and compressing the second model may depend on the situation. For example, the method of quantizing and compressing may be any algorithm such as the post-training quantization algorithm (PTQ) and the offline quantization algorithm Brecq, which is not limited in the embodiments of the present application.

[0073] As an optional embodiment, saving the third model as a target model for deployment on the terminal device includes: saving the third model in a cloud-based universal format; determining a model reasoning format corresponding to the terminal device; and converting the third model from the cloud-based universal format to a model reasoning format to obtain a target model for deployment on the terminal device. This can improve the compatibility of the target model with the terminal device, which is conducive to improving the operating efficiency of the terminal device.

[0074] Specifically, the cloud general format may be onnx or other formats, and the model inference format may be tflite, mnn or other formats. In addition, optionally, the target model may be tested on a cloud real machine, which is conducive to further adjustment of the target model.

[0075] As an optional embodiment, the first model is compressed to obtain the second model, including: constructing a super network based on the first model; adjusting at least one of the convolution kernel size, depth, and width of the super network, and training the super network after each adjustment to obtain multiple training results; and pruning and compressing the super network based on the multiple training results to obtain the second model. In this way, the number of network parameters can be reduced to obtain a lightweight model.

[0076] Among them, hypernetworks represent small-scale networks used to generate parameters of larger-scale networks. In this process, the role of the main network is the same as any other neural network, mapping input samples to corresponding target values. The hypernetwork is used to receive a series of values ​​containing the main network parameter structure information as input and then generate the parameters of a certain layer of the main network. The idea of ​​hypernetworks originated from evolutionary computing: using smaller networks to generate parameters of larger networks so that the search space is limited to a smaller space. The parameters of the hypernetwork and the main network are trained in an end-to-end manner.

[0077] Specifically, it may also include: training a super-network. Furthermore, optionally, adjusting at least one of the convolution kernel size, depth, and width of the super-network, and training the super-network after each adjustment to obtain multiple training results, including: using the largest kernel size as the size of the original network, keeping the depth and width at the maximum, and then fine-tuning the model (Fine Tune) to save computing resources and computing time, and improve computing efficiency and accuracy; further, adjusting the depth and keeping the width at the maximum, and then fine-tuning the model (Fine Tune) to achieve the maximum network (sub-network); further, adjusting the width; wherein, optionally, the kernel size can be any one of {3,5,7}.

[0078] Optionally, a distillation method can be used in the process of training each adjusted hypernetwork so that the second model uses the labels (soft labels) provided by the hypernetwork, wherein the loss function corresponding to the distillation method is Loss=Lossh ard +λLoss 2oft Among them, Lossh ard It is generated based on the labels in the sample dataset, Loss 2oft It is generated based on the non-original labels related to the subnet, and λ is the weight set according to the requirements.

[0079] See also Figure 5 , Figure 5 The following schematically shows a flow chart of a model generation method according to another embodiment of the present application. Figure 5 As shown, the model generation method includes: steps S500 to S518.

[0080] Step S500: In response to the customization operation, determine the customized terminal device model, model task, and model constraint condition.

[0081] Step S502: In response to a data source creation request, the data source type is displayed, and in response to a selection operation on the data source type, a target data source type is determined, and then, based on the target data source type, a sample data set is imported into a model training platform.

[0082] Step S504: Perform statistical analysis on the sample data set through the model training platform to obtain statistical analysis results, and display the statistical analysis results in response to a viewing operation on the statistical analysis results.

[0083] Step S506: In response to an automatic balancing request for statistical analysis results, if the sample data set is an audio data set, the data volume corresponding to each label in the sample data set is balanced by at least one processing method selected from the group consisting of noise enhancement, audio splicing, time shift enhancement, pitch conversion, speed adjustment, volume adjustment, mixed background sound, white noise supplement, moving audio, and stretching audio; if the sample data set is an image data set, the data volume corresponding to each label in the sample data set is balanced by at least one processing method selected from the group consisting of horizontal flipping, vertical flipping, rotation, scaling, cropping, translation, Gaussian noise, generative adversarial network, and screenshot; if the sample data set is a text data set, the data volume corresponding to each label in the sample data set is balanced by at least one processing method selected from the group consisting of syntax tree structure replacement, paragraph extraction, sequence generation, generative adversarial network, and pre-trained language model.

[0084] Step S508: Determine the hardware information corresponding to the terminal device model from the hardware information library, and select the model architecture based on binary gates based on the hardware information and the model task to obtain multiple reference models; wherein the binary gates are used to limit the architecture parameters based on which the model architecture is selected each time.

[0085] Step S510: training a plurality of reference models based on the sample data set to obtain a plurality of intermediate models.

[0086] Step S512: Determine a first model from a plurality of intermediate models based on model constraints.

[0087] Step S514: construct a hypernetwork based on the first model; adjust at least one of the convolution kernel size, depth, and width of the hypernetwork, and train the hypernetwork after each adjustment to obtain multiple training results, and prune and compress the hypernetwork based on the multiple training results to obtain a second model; wherein the parameter amount of the second model is less than the parameter amount of the first model.

[0088] Step S516: quantize and compress the second model to obtain a third model; wherein the storage space occupied by the third model is smaller than the storage space occupied by the second model.

[0089] Step S518: Save the third model in a universal cloud format, and determine the model inference format corresponding to the terminal device, and then convert the third model from the universal cloud format to the model inference format to obtain a target model for deployment on the terminal device; wherein the target model and the third model correspond to different storage formats.

[0090] It should be noted that steps S500 to S518 are Figure 1 The steps and their embodiments shown correspond to each other. For the specific implementation of steps S500 to S518, please refer to Figure 1 The steps and embodiments shown are not described in detail here.

[0091] It can be seen that implementation Figure 5 The method shown can take into account the hardware differences between devices and provide custom functions. For users, they can select terminal device models, model tasks, and model constraints through custom operations. The present application can generate personalized target models for terminal device models, model tasks, and model constraints. The target model is more compatible with the terminal device and is easier to achieve reasonable use of terminal device resources. In addition, since the present application can automatically generate personalized target models for terminal devices, the present application simultaneously realizes automation and personalization in the model generation process.

[0092] See also Figure 6 , Figure 6 The structure block diagram of the model generation device in one embodiment of the present application is schematically shown. Figure 6 As shown, the model generating device 600 may include the following units.

[0093] A custom information determination unit 601 is used to determine a custom terminal device model, model task, and model constraint condition in response to a custom operation;

[0094] The model generation unit 602 is used to generate a target model for deployment on a terminal device according to the terminal device model, model tasks, and model constraints.

[0095] It can be seen that implementation Figure 6 The device shown can take into account the hardware differences between devices and provide custom functions. For users, they can select terminal device models, model tasks, and model constraints through custom operations. The present application can generate personalized target models for terminal device models, model tasks, and model constraints. The target model is more compatible with the terminal device and is easier to achieve reasonable use of terminal device resources. In addition, since the present application can automatically generate personalized target models for terminal devices, the present application simultaneously realizes automation and personalization in the model generation process.

[0096] As an optional embodiment, the model generation unit 602 generates a target model for deployment on the terminal device according to the terminal device model, the model task, and the model constraint condition, including:

[0097] In response to the setting operation, determining the set training reference conditions; wherein the model training reference conditions include at least one of a neural network block, a model morphology, a model input size, and a model output size;

[0098] Generate a target model for deployment on the terminal device based on the terminal device model, model tasks, model constraints, and model training reference conditions.

[0099] It can be seen that by implementing this optional embodiment, a target model that better meets user needs and is adapted to terminal devices can be generated based on conditions in more dimensions.

[0100] As an optional embodiment, the model generation unit 602 generates a target model for deployment on the terminal device according to the terminal device model, the model task, and the model constraint condition, including:

[0101] Obtain a sample data set, and generate a first model according to the sample data set, the model task, and the model constraints;

[0102] The first model is saved as a target model for deployment on a terminal device.

[0103] It can be seen that the implementation of this optional embodiment can not only realize the model generation based on sample data sets, model tasks, and model constraints, but also save the model as a target model suitable for terminal device reasoning, which can save manual design costs and improve the generation efficiency of personalized models.

[0104] As an optional embodiment, the model generation unit 602 obtains a sample data set, including:

[0105] In response to a data source creation request, display the data source type;

[0106] In response to a selection operation on a data source type, determining a target data source type;

[0107] Import the sample dataset into the model training platform based on the target data source type.

[0108] It can be seen that implementing this optional embodiment can facilitate users to customize data source types, and then import sample data sets into the model training platform based on the defined data source types, which can improve interactivity with users.

[0109] As an optional embodiment, it also includes:

[0110] The statistical unit is used to perform statistical analysis on the sample data set through the model training platform to obtain statistical analysis results;

[0111] A display unit, configured to display the statistical analysis result in response to a viewing operation on the statistical analysis result;

[0112] The data enhancement unit is used to perform a data enhancement operation to balance the amount of data corresponding to each label in the sample data set in response to an automatic balancing request for the statistical analysis result.

[0113] It can be seen that implementing this optional embodiment can provide users with automated data analysis functions and display the results of data analysis to users so that users can decide whether to activate the data enhancement function, thereby improving interactivity with users.

[0114] As an optional embodiment, the model generation unit 602 performs statistical analysis on the sample data set through the model training platform to obtain statistical analysis results, including:

[0115] The model training platform performs at least one of deviation analysis, histogram analysis, scatter plot analysis, and table statistics on the sample data set to obtain statistical analysis results.

[0116] It can be seen that the implementation of this optional embodiment can provide users with more diverse data statistics functions to facilitate users' subsequent data processing and other operations.

[0117] As an optional embodiment, the model generation unit 602 performs a data enhancement operation to balance the amount of data corresponding to each label in the sample data set, including:

[0118] If the sample data set is an audio data set, the amount of data corresponding to each label in the sample data set is balanced by at least one processing method selected from the group consisting of noise enhancement, audio splicing, time shift enhancement, pitch conversion, speed adjustment, volume adjustment, background sound mixing, white noise supplementation, audio movement, and audio stretching;

[0119] If the sample data set is an image data set, the amount of data corresponding to each label in the sample data set is balanced by at least one of horizontal flipping, vertical flipping, rotation, scaling, cropping, translation, Gaussian noise, generative adversarial network, and screenshot processing;

[0120] If the sample data set is a text data set, the amount of data corresponding to each label in the sample data set is balanced through at least one of grammatical tree structure replacement, paragraph truncation, sequence generation, generative adversarial network, and pre-trained language model.

[0121] It can be seen that the implementation of this optional embodiment can balance the data volume of each label in the sample data set and avoid the situation where the sample volume of one label is too small, which is conducive to improving the training effect of the model and improving the model training accuracy.

[0122] As an optional embodiment, the model generation unit 602 generates a first model according to the sample data set, the model task, and the model constraint condition, including:

[0123] Determine the hardware information corresponding to the terminal device model from the hardware information database;

[0124] Based on the hardware information and model tasks, a binary gate-based model architecture is selected to obtain a variety of reference models; wherein the binary gate is used to limit the architecture parameters based on which the model architecture is selected each time;

[0125] Based on the sample data set, multiple reference models are trained respectively to obtain multiple intermediate models;

[0126] A first model is determined from a plurality of intermediate models based on the model constraints.

[0127] It can be seen that by implementing this optional embodiment, a model architecture can be selected based on the set binary gate to determine an architecture suitable for participating in model training, and this architecture is more suitable for a specified terminal device, which can be conducive to more efficiently generating a personalized model for deployment on a terminal device.

[0128] As an optional embodiment, the model generation unit 602 saves the first model as a target model for deployment on the terminal device, including:

[0129] Performing network compression on the first model to obtain a second model; wherein the number of parameters of the second model is smaller than the number of parameters of the first model;

[0130] Quantizing and compressing the second model to obtain a third model; wherein the storage occupied by the third model is less than the storage occupied by the second model;

[0131] The third model is saved as a target model for deployment on a terminal device; wherein the target model and the third model correspond to different storage formats.

[0132] It can be seen that the implementation of this optional embodiment can reduce the size of the model that needs to be deployed on the terminal device based on network compression and quantization compression, avoid affecting the performance of the terminal device to improve the operating efficiency of the terminal device, and improve the model deployment efficiency.

[0133] As an optional embodiment, the model generation unit 602 saves the third model as a target model for deployment on the terminal device, including:

[0134] Save the third model in a common format on the cloud;

[0135] Determine the model inference format corresponding to the terminal device;

[0136] The third model is converted from a cloud general format into a model reasoning format to obtain a target model for deployment on a terminal device.

[0137] It can be seen that implementing this optional embodiment can improve the compatibility between the target model and the terminal device, which is beneficial to improving the operating efficiency of the terminal device.

[0138] As an optional embodiment, the model generation unit 602 performs network compression on the first model to obtain a second model, including:

[0139] constructing a hypernetwork based on the first model;

[0140] Adjust at least one of the convolution kernel size, depth, and width of the hypernetwork, and train the hypernetwork after each adjustment to obtain multiple training results;

[0141] The super network is pruned and compressed based on multiple training results to obtain a second model.

[0142] It can be seen that implementing this optional embodiment can reduce the number of network parameters and obtain a lightweight model.

[0143] It should be noted that, although several modules or units of the equipment for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into being embodied by multiple modules or units.

[0144] Since the various functional modules of the model generation device of the example embodiment of the present application correspond to the steps of the example embodiment of the model generation device described above, for details not disclosed in the embodiment of the device of the present application, please refer to the embodiment of the model generation device described above in the present application.

[0145] See also Figure 7 , Figure 7 A schematic diagram of the structure of a computer system suitable for implementing an electronic device of an embodiment of the present application is shown.

[0146] It should be noted that Figure 7 The computer system 700 of the electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0147] like Figure 7As shown, computer system 700 includes a central processing unit (CPU) 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage part 708 into a random access memory (RAM) 703. Various programs and data required for system operation are also stored in RAM 703. CPU 701, ROM 702 and RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to bus 704.

[0148] The following components are connected to the I / O interface 705: an input section 706 including a keyboard, a mouse, etc.; an output section 707 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 708 including a hard disk, etc.; and a communication section 709 including a network interface card such as a LAN card, a modem, etc. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the I / O interface 705 as needed. A removable medium 711, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 710 as needed, so that a computer program read therefrom is installed into the storage section 708 as needed.

[0149] In particular, according to an embodiment of the present application, the process described below with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication section 709, and / or installed from a removable medium 711. When the computer program is executed by a central processing unit (CPU) 701, various functions defined in the method and apparatus of the present application are executed.

[0150] As another aspect, the present application also provides a computer-readable medium, which may be included in the electronic device described in the above embodiment; or may exist independently without being assembled into the electronic device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by an electronic device, the electronic device implements the method in the above embodiment.

[0151] It should be noted that the computer-readable medium shown in the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0152] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the above-mentioned module, program segment or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0153] The units involved in the embodiments described in this application may be implemented by software or hardware, and the units described may also be set in a processor. The names of these units do not, in some cases, constitute limitations on the units themselves.

[0154] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary technical means in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the claims.

Claims

1. A model generation method, It is characterized in that include: In response to the custom operation, determining a custom terminal device model, model task, and model constraint condition; A target model for deployment on the terminal device is generated according to the terminal device model, the model task, and the model constraint condition.

2. The method according to claim 1, It is characterized in that Generating a target model for deployment on a terminal device according to the terminal device model, the model task, and the model constraint condition, including: In response to the setting operation, determining the set training reference conditions; wherein the model training reference conditions include at least one of a neural network block, a model morphology, a model input size, and a model output size; A target model for deployment on a terminal device is generated according to the terminal device model, the model task, the model constraint conditions, and the model training reference conditions.

3. The method according to claim 1, It is characterized in that Generating a target model for deployment on a terminal device according to the terminal device model, the model task, and the model constraint condition, including: Acquire a sample data set, and generate a first model according to the sample data set, the model task, and the model constraint condition; The first model is saved as a target model for deployment on a terminal device.

4. The method according to claim 3, It is characterized in that Get a sample dataset, including: In response to a data source creation request, display the data source type; In response to a selection operation on the data source type, determining a target data source type; The sample data set is imported into the model training platform based on the target data source type.

5. The method according to claim 3, It is characterized in that Also includes: Performing statistical analysis on the sample data set through the model training platform to obtain statistical analysis results; In response to a viewing operation on the statistical analysis result, displaying the statistical analysis result; In response to an automatic balancing request for the statistical analysis result, a data augmentation operation is performed to balance the amount of data corresponding to each label in the sample data set.

6. The method according to claim 5, It is characterized in that The sample data set is statistically analyzed by the model training platform to obtain statistical analysis results, including: The model training platform performs at least one of deviation analysis, histogram analysis, scatter plot analysis, and table statistics on the sample data set to obtain a statistical analysis result.

7. The method according to claim 5, It is characterized in that Performing data augmentation operations to balance the amount of data corresponding to each label in the sample data set includes: If the sample data set is an audio data set, the amount of data corresponding to each label in the sample data set is balanced by at least one processing method selected from the group consisting of noise enhancement, audio splicing, time shift enhancement, pitch conversion, speed adjustment, volume adjustment, mixed background sound, white noise supplementation, audio movement, and audio stretching; If the sample data set is an image data set, the amount of data corresponding to each label in the sample data set is balanced by at least one processing method selected from horizontal flipping, vertical flipping, rotation, scaling, cropping, translation, Gaussian noise, generative adversarial network, and screenshot; If the sample data set is a text data set, the amount of data corresponding to each label in the sample data set is balanced by at least one processing method selected from the group consisting of syntax tree structure replacement, paragraph extraction, sequence generation, generative adversarial network, and pre-trained language model.

8. The method according to claim 3, It is characterized in that Generating a first model according to the sample data set, the model task, and the model constraint condition includes: Determine the hardware information corresponding to the terminal device model from the hardware information library; Based on the hardware information and the model task, a model architecture selection based on a binary gate is performed to obtain a plurality of reference models; wherein the binary gate is used to limit the architecture parameters based on which the model architecture is selected each time; Based on the sample data set, the multiple reference models are trained respectively to obtain multiple intermediate models; A first model is determined from the plurality of intermediate models based on the model constraints.

9. The method according to claim 3, It is characterized in that Saving the first model as a target model for deployment on a terminal device includes: Performing network compression on the first model to obtain a second model; wherein the number of parameters of the second model is smaller than the number of parameters of the first model; Quantizing and compressing the second model to obtain a third model; wherein the storage occupied by the third model is less than the storage occupied by the second model; The third model is saved as a target model for deployment on a terminal device; wherein the target model and the third model correspond to different storage formats.

10. The method according to claim 9, It is characterized in that Saving the third model as a target model for deployment on a terminal device includes: Saving the third model in a cloud-based universal format; Determine the model inference format corresponding to the terminal device; The third model is converted from the cloud general format to the model inference format to obtain a target model for deployment on a terminal device.

11. The method according to claim 9, It is characterized in that Performing network compression on the first model to obtain a second model includes: Building a hypernetwork based on the first model; Adjusting at least one of the convolution kernel size, depth, and width of the hypernetwork, and training the hypernetwork after each adjustment to obtain multiple training results; The super network is pruned and compressed based on the multiple training results to obtain a second model.

12. A model generating device, It is characterized in that include: A custom information determination unit, used to determine a custom terminal device model, model task, and model constraint condition in response to a custom operation; The model generation unit is used to generate a target model for deployment on the terminal device according to the terminal device model, the model task, and the model constraint conditions.

13. A computer-readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 11 is implemented.

14. An electronic device, It is characterized in that include: processor; as well as A memory, configured to store executable instructions of the processor; The processor is configured to perform the method of any one of claims 1-11 by executing the executable instructions.

Citation Information

Patent Citations

  • Data access method, device, equipment and medium

    CN110555072A

  • Space domain limited pixel target detection system and method fusing neural network space-time characteristics

    CN114022759A

  • Model acquisition method, electronic equipment and storage medium

    CN114662700A

  • Automatically and efficiently generating search spaces for neural network

    US20220398450A1

  • Pruning and quantization compression method and system for super-resolution network, and medium

    WO2022262660A1