Method for obtaining neural network model, image processing method and device

Abstract

The application discloses a method for obtaining a neural network model, an image processing method and device in the field of artificial intelligence. The method for obtaining the neural network model comprises the following steps: obtaining a pre-trained super network model, the pre-trained super network model being obtained based on a source data set; obtaining a target data set, the target data set corresponding to the same task as the source data set; performing transfer learning on the pre-trained super network model based on the target data set to obtain a super network model after transfer learning; and searching for a sub network model in the super network model after transfer learning to obtain a target neural network model. The method can reduce the training cost in the process of obtaining the required neural network model and improve the performance of the neural network model.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and more specifically, to a method for acquiring a neural network model, an image processing method, and an apparatus. Background Technology

[0002] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a branch of computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to have perception, reasoning, and decision-making capabilities. Research in the field of AI includes robotics, natural language processing, computer vision, decision-making and reasoning, human-computer interaction, recommendation and search, and fundamental AI theories.

[0003] With the rapid development of artificial intelligence technology, the performance of neural network models (e.g., convolutional neural network models) has been continuously improved, and neural network models have achieved great success in the processing and analysis of various media signals such as images, videos, and audio. Taking image recognition as an example, deep neural network models outperform traditional computer vision methods by a significant margin. However, training a good deep neural network model requires extensive expert experience. In recent years, the use of automated machine learning (AutoML) technology to automatically search for neural network models has gradually become a hot topic in the field of computer vision. AutoML can obtain neural network models that are superior to those designed manually. However, the training resources required for AutoML, such as training machines and training data, are often much larger than those for ordinary neural network models, and the training cost of AutoML is also much higher than that of ordinary neural network models. In small data scenarios, due to insufficient training data, AutoML often struggles to directly train excellent neural network models.

[0004] Therefore, how to obtain the required neural network model through AutoML has become an urgent problem to be solved. Summary of the Invention

[0005] This application provides a method, image processing method, and apparatus for obtaining a neural network model, which reduces training costs and improves the performance of the neural network model during the process of obtaining the required neural network model.

[0006] Firstly, a method for obtaining a neural network model is provided, the method comprising: obtaining a pre-trained hypernetwork model, the pre-trained hypernetwork model being trained based on a source dataset; obtaining a target dataset, the task corresponding to the target dataset being the same as the task corresponding to the source dataset; performing transfer learning on the pre-trained hypernetwork model based on the target dataset to obtain a hypernetwork model after transfer learning; and searching for subnetwork models in the hypernetwork model after transfer learning to obtain a target neural network model.

[0007] The source dataset can be a large dataset, which ensures that the hypernetwork model is trained sufficiently and can obtain a hypernetwork model with higher accuracy.

[0008] It should be noted that the source dataset can be a dataset related to the task that the target neural network model needs to perform. That is, the sub-network models in the pre-trained supernetwork model perform the same task as the target neural network model. For example, both are used for image classification; or both are used for image segmentation; or both are used for object detection.

[0009] For example, when the target neural network model is used for image classification, the source dataset can be the publicly available ImageNet dataset.

[0010] The target dataset can be a dataset input by the user or a dataset obtained from other devices.

[0011] Transfer learning on a pre-trained hypernetwork model based on a target dataset can be used to fine-tune a pre-trained hypernetwork model based on the target dataset.

[0012] Transfer learning on a pre-trained hypernetwork model refers to transferring the weights of the pre-trained hypernetwork model.

[0013] The target neural network model refers to a neural network model whose performance metrics meet the target performance metrics. In other words, the target sub-network model can be found by searching within the supernetwork model after transfer learning, and the target neural network model is determined based on this target sub-network model. This target sub-network model can be a sub-network model whose performance metrics meet the target performance metrics. The target sub-network model can be a single sub-network model or multiple sub-network models.

[0014] The performance metrics of a subnetwork model can include its inference accuracy, hardware overhead, or inference time. The target performance metrics can include target accuracy, target overhead, or target inference time.

[0015] For example, searching for sub-network models in the super-network model after transfer learning can be done by using a reinforcement learning algorithm to search for sub-network models in the super-network model after transfer learning to obtain the target neural network model.

[0016] In this embodiment, by transferring the pre-trained hypernetwork model to the target dataset, a high-performance hypernetwork model can be obtained even when the target dataset is small, thereby enabling the search for the target neural network model. This empowers applications in small data scenarios and greatly improves the accuracy of AutoML in such scenarios.

[0017] At the same time, to meet different user needs, such as user cost / accuracy requirements, a neural network model that meets the user's needs can be obtained by searching the sub-network models in the hypernetwork model, and adapted to the target dataset to meet the user's needs, such as meeting the user's cost / accuracy requirements.

[0018] Meanwhile, the weights of the hypernetic network model are shared between different datasets. Both the source and target datasets are datasets related to the same task, which enables efficient transfer learning of AutoML. During transfer, only the weights of the hypernetic network model are fine-tuned without adjusting the structure of the hypernetic network model. This can greatly improve the transfer efficiency of AutoML and reduce the training time required by at least one order of magnitude, or even reach the training time of ordinary neural network models.

[0019] Furthermore, the transfer learning time of the hypernetwork model provided in this embodiment is close to that of a regular neural network model. In other words, compared to methods that obtain the target neural network model through transfer learning of a regular neural network model, the method for obtaining the neural network model in this embodiment better meets the user's refined cost / accuracy requirements under the same training time. It achieves a more accurate target neural network model with the same overhead.

[0020] Furthermore, for the same task, such as image classification, when users need multiple neural network models, there is no need to design and train neural network models separately for each deployment scheme or user needs. Only one supernetwork model needs to be trained. The weights of the supernetwork model are shared or transferred to different datasets, resulting in neural network models that meet the different overhead / accuracy requirements of users, which greatly reduces training costs.

[0021] In conjunction with the first aspect, in some implementations of the first aspect, the pre-trained hypernetwork model is trained using a progressive shrinkage method.

[0022] Specifically, training a supernetwork model using the progressive shrinkage method can include: first training the largest subnetwork model, and then progressively training subnetwork models with variable convolutional kernels, subnetwork models with variable number of layers, and subnetwork models with variable number of channels.

[0023] Among them, the largest subnetwork model refers to the subnetwork model with the largest convolutional kernel, the largest number of layers (depth), and the largest number of channels (width) in the supernetwork model.

[0024] Specifically, subnetwork models with variable convolutional kernels, variable number of layers, and variable number of channels can be trained by performing knowledge distillation on the maximum subnetwork model.

[0025] Because sub-network models share weights, interference may occur between different sub-network models during supernetwork model training. According to the scheme in this embodiment, training with a progressive shrinkage algorithm reduces the mutual influence between sub-network models of different sizes during training, resulting in a supernetwork model that supports various architecture settings. For example, various architecture settings include sub-network models with different numbers of layers, different numbers of channels, and different convolutional kernel sizes. After the supernetwork model is trained, suitable sub-network models can be selected from it without additional training of the searched sub-network models, or in other words, the sub-network models do not need to be retrained, and the accuracy of the sub-network models can still meet the pre-training requirements. Furthermore, during the training of the supernetwork model, it is not necessary to train each sub-network model independently; the sub-network models in this supernetwork model can achieve an accuracy similar to independently trained sub-network models.

[0026] In conjunction with the first aspect, in certain implementations of the first aspect, transfer learning is performed on a pre-trained hypernetwork model based on a target dataset to obtain a transferred-learned hypernetwork model, including: selecting a sub-network model from the pre-trained hypernetwork model; calculating the weight gradient of the sub-network model based on the target dataset; updating the weights of the sub-network model based on the weight gradient of the sub-network model to obtain an updated sub-network model; obtaining an updated hypernetwork model based on the updated sub-network model; repeating the above steps until the updated hypernetwork model meets a termination condition to obtain the transferred-learned hypernetwork model; wherein, the termination condition includes at least one of the following: the number of repetitions is greater than or equal to the number of first iterations; the inference accuracy of the updated hypernetwork model is greater than or equal to the first inference accuracy.

[0027] In other words, in each training iteration, only the sub-network model of a single path is activated, or it can be understood as selecting only one sub-network model from the supernetwork model, updating the weights of that sub-network model, and continuing the iteration until training is complete. During each training iteration, only the weights of the selected sub-network model are activated and updated.

[0028] According to the scheme of this application embodiment, by using a single-path algorithm to transfer the supernetwork model, subnetwork models can be sampled uniformly and trained, thereby improving the training effect. Furthermore, it can reduce memory space and achieve efficient training.

[0029] In conjunction with the first aspect, in some implementations of the first aspect, the step of performing transfer learning on the pre-trained hypernetwork model based on the target dataset to obtain a transferred-learned hypernetwork model includes: selecting N from the pre-trained hypernetwork model. b Each sub-network model calculates the N based on the target dataset. b The weight gradients of each sub-network model, based on the N... b The weight gradient update of the N sub-network model b The weights of each sub-network model are used to obtain the updated super-network model, N. b It is a positive integer;

[0030] Repeat the above steps until the updated hypernetwork model meets the termination condition to obtain the hypernetwork model after transfer learning. The termination condition includes at least one of the following: the number of repetitions is greater than or equal to the number of the first iteration; the inference accuracy of the updated hypernetwork model is greater than or equal to the first inference accuracy.

[0031] Each time a sub-network model is selected, only single-path sub-network models are activated; or, this can be understood as selecting only one sub-network model each time a sub-network model is chosen from the super-network model. N was selected. b The sub-network model can also be understood as selecting N. b Secondary child network model.

[0032] Update N b The weights of the sub-network model are also the weights for updating the super-network model.

[0033] For example, updating the weights of a hypernetwork model can include: subtracting N from the current weights of the hypernetwork model. b The weight gradients of the sub-network models are used to update the weights of the super-network model.

[0034] Alternatively, updating the weights of the hypernetwork model can include: subtracting N from the current weights of the hypernetwork model. b The weights of the supernetwork model are updated by multiplying the weight gradient of each subnetwork model by the learning rate.

[0035] For example, the inference accuracy of the updated hypernetwork model can be the inference accuracy of at least one subnetwork model in the hypernetwork model.

[0036] Since sub-network models may share weights, updating the weights of the current sub-network model based on the weight gradients calculated in each backpropagation will interfere with other sub-network models that share weights. According to the scheme of this application embodiment, multiple forward and backward propagations are performed in each iteration, accumulating the weight gradients of multiple sub-network models in one iteration, and updating the weights of the supernetwork model only once. This reduces mutual interference between different sub-network models, improves the accuracy of the supernetwork model, and increases its training speed.

[0037] In conjunction with the first aspect, in certain implementations of the first aspect, a sub-network model is searched within the transferred supernetwork model to obtain the target neural network model, including:

[0038] Step 1: Determine n first sub-network models based on the supernetwork model after transfer learning, where n is an integer greater than 1;

[0039] Step 2: Adjust the structure of the n first sub-network models to obtain n second sub-network models;

[0040] Step 3: Select n third sub-network models from the n first sub-network models and the n second sub-network models, and use the n third sub-network models as the n first sub-network models in Step 2;

[0041] Repeat steps two to three above until the n third sub-network models satisfy the search termination condition, which includes at least one of the following: the number of repetitions is greater than or equal to the number of second iterations, or the inference accuracy of at least p of the n third sub-network models is greater than or equal to the target accuracy.

[0042] The target neural network model is determined based on the n third sub-network models.

[0043] For example, n sub-network models can be extracted from the supernetwork model after transfer learning. These n sub-network models are the n first sub-network models. These n first sub-network models can be considered as a population.

[0044] For example, adjusting the structure of n first sub-network models can be achieved through operations such as crossover and mutation.

[0045] In conjunction with the first aspect, in some implementations of the first aspect, determining n first sub-network models based on the supernetwork model after transfer learning includes: selecting n fourth sub-network models from the supernetwork model after transfer learning; obtaining the hardware overhead of the n fourth sub-network models on the target device; and adjusting the structure of the n fourth sub-network models based on the hardware overhead to obtain the n first sub-network models.

[0046] Optionally, selecting n fourth sub-network models in the transferred supernetwork model can be done by randomly selecting n fourth sub-network models.

[0047] Optionally, adjusting the subnetwork model structure based on the hardware overhead of the n fourth subnetwork models on the target device can include: adjusting the subnetwork model structure according to the probability of subnetwork model structure adjustment, wherein the adjusted subnetwork model can meet the target overhead. The probability of subnetwork model structure adjustment is determined based on the hardware overhead of the subnetwork model.

[0048] For example, for a subnetwork model with high hardware overhead, the probability of adjusting the current subnetwork model to a smaller subnetwork model is greater than the probability of adjusting the current subnetwork model to a larger subnetwork model. For a subnetwork model with low hardware overhead, the probability of adjusting the current subnetwork model to a larger subnetwork model is greater than the probability of adjusting the current subnetwork model to a smaller subnetwork model. The hardware overhead can be determined relative to a target overhead. For example, a subnetwork model with an overhead greater than the target overhead can be considered a subnetwork model with high hardware overhead, and a subnetwork model with an overhead less than the target overhead can be considered a subnetwork model with low hardware overhead. Alternatively, the hardware overhead can also be determined relative to other benchmarks, and this embodiment of the application does not limit this.

[0049] For example, the target device may include a GPU or an NPU.

[0050] According to the scheme of the embodiments of this application, a heuristic search method is adopted to perceive the hardware overhead of the sub-network model on the target device, adjust the structure of the sub-network model based on the hardware overhead, and then perform a search, so that the final sub-network model can meet the target overhead.

[0051] Secondly, an image processing method is provided, which includes: acquiring an image to be processed; processing the image to be processed using a target neural network model to obtain the processing result of the image to be processed; wherein, the target neural network model is obtained by searching for a sub-network model in a hypernetwork model, the hypernetwork model is obtained by transfer learning of a pre-trained hypernetwork model based on the target dataset, the pre-trained hypernetwork model is trained based on the source dataset, and the task corresponding to the target dataset is the same as the task corresponding to the source dataset.

[0052] In this application, since the target neural network model is obtained using the method described in the first aspect above, it closely matches or closely aligns with the application requirements of neural network models. Using such a neural network model for image classification can achieve better image classification results (e.g., more accurate classification results, etc.). Even with a small target dataset, a high-performance hypernetwork model can be obtained, enabling applications in small data scenarios and greatly improving the accuracy of AutoML in small data scenarios, thereby obtaining a target neural network model that meets the needs of different users.

[0053] In conjunction with the second aspect, in some implementations of the second aspect, the pre-trained supernetwork model is obtained by training through a progressive shrinkage method.

[0054] In conjunction with the second aspect, in some implementations of the second aspect, the hypernetwork model is obtained by transfer learning from a pre-trained hypernetwork model based on the target dataset, including: selecting a sub-network model from the pre-trained hypernetwork model, calculating the weight gradient of the sub-network model based on the target dataset, updating the weights of the sub-network model based on the weight gradient of the sub-network model to obtain an updated sub-network model, and obtaining an updated hypernetwork model based on the updated sub-network model; repeating the above steps until the updated hypernetwork model meets the termination condition; wherein, the termination condition includes at least one of the following: the number of repetitions is greater than or equal to the number of first iterations; the inference accuracy of the updated hypernetwork model is greater than or equal to the first inference accuracy.

[0055] In conjunction with the second aspect, in some implementations of the second aspect, the hypernetwork model is obtained by transfer learning from a pre-trained hypernetwork model based on the target dataset, including: the hypernetwork model is obtained by selecting N from the pre-trained hypernetwork model. b Each sub-network model calculates the N based on the target dataset. b The weight gradients of each sub-network model, based on the N... b The weight gradient update of the N sub-network model b The weights of each sub-network model are used to obtain the updated N. b A sub-network model, based on the updated N b The updated hypernetwork model is obtained from the N subnetwork models. b The value is a positive integer; repeat the above steps until the updated hypernetwork model meets the termination condition, wherein the termination condition includes at least one of the following: the number of repetitions is greater than or equal to the number of the first iteration; the inference accuracy of the updated hypernetwork model is greater than or equal to the first inference accuracy.

[0056] In conjunction with the second aspect, in some implementations of the second aspect, the target neural network model is obtained by searching for sub-network models in the supernetwork model, including: determining n first sub-network models based on the supernetwork model, where n is an integer greater than 1; adjusting the structure of the n first sub-network models to obtain n second sub-network models; selecting n third sub-network models from the n first sub-network models and the n second sub-network models, and updating the n third sub-network models to the n first sub-network models; repeating the above steps until the n third sub-network models satisfy the search termination condition; determined based on the n third sub-network models; wherein the search termination condition includes at least one of the following: the number of repetitions is greater than or equal to the number of second iterations, or the inference accuracy of at least p third sub-network models among the n third sub-network models is greater than or equal to the target accuracy.

[0057] In conjunction with the second aspect, in some implementations of the second aspect, determining n first sub-network models based on the supernetwork model includes: selecting n fourth sub-network models in the supernetwork model; obtaining the hardware overhead of the n fourth sub-network models on the target device; and adjusting the structure of the n fourth sub-network models based on the hardware overhead to obtain n first sub-network models.

[0058] Thirdly, an apparatus for acquiring a neural network model is provided, the apparatus comprising a module or unit for performing the methods described in the first aspect and any implementation thereof.

[0059] Fourthly, an image processing apparatus is provided, comprising a module or unit for performing the methods described in the second aspect and any implementation thereof.

[0060] It should be understood that the extensions, limitations, interpretations and descriptions of the relevant content in the first aspect above also apply to the same content in the second, third and fourth aspects.

[0061] Fifthly, an apparatus for acquiring a neural network model is provided, the apparatus comprising: a memory for storing a program; and a processor for executing the program stored in the memory, wherein when the program stored in the memory is executed, the processor is configured to execute the method of the first aspect and any implementation thereof.

[0062] The processor mentioned in the fifth aspect above can be a central processing unit (CPU) or a combination of a CPU and a neural network model computation processor. This neural network model computation processor can include a graphics processing unit (GPU), a neural network processing unit (NPU), and a tensor processing unit (TPU), etc. The TPU is a dedicated integrated circuit from Google, designed entirely for machine learning and artificial intelligence accelerators.

[0063] In a sixth aspect, an image processing apparatus is provided, comprising: a memory for storing a program; and a processor for executing the program stored in the memory, wherein when the program stored in the memory is executed, the processor is configured to execute the method of the second aspect and any implementation thereof.

[0064] The processor mentioned in the sixth aspect above can be a central processing unit (CPU) or a combination of a CPU and a neural network model computation processor. This neural network model computation processor can include a graphics processing unit (GPU), a neural network model processor, and a tensor processor, among others. The TPU is a Google-designed application-specific integrated circuit (ASIC) for a fully customized AI accelerator designed for machine learning.

[0065] A seventh aspect provides a computer-readable medium storing program code for execution by a device, the program code including methods for performing any implementation of the first or second aspect.

[0066] Eighthly, a computer program product containing instructions is provided, which, when run on a computer, causes the computer to perform the method in any one of the implementations of the first or second aspect described above.

[0067] Ninth aspect, a chip is provided, the chip including a processor and a data interface, the processor reading instructions stored in a memory through the data interface and executing the method in any one of the implementations of the first or second aspect described above.

[0068] Optionally, as one implementation, the chip may further include a memory storing instructions, and the processor is used to execute the instructions stored in the memory. When the instructions are executed, the processor is used to execute the method in either the first aspect or the second aspect.

[0069] The aforementioned chip can be a field-programmable gate array (FPGA) or an application-specific integrated circuit (ASIC). Attached Figure Description

[0070] Figure 1 This is a schematic diagram of an artificial intelligence main framework provided in an embodiment of this application;

[0071] Figure 2 A schematic diagram of a system architecture provided in an embodiment of this application;

[0072] Figure 3 This is a schematic diagram of the structure of a convolutional neural network model provided in an embodiment of this application;

[0073] Figure 4 This is a schematic diagram of another convolutional neural network model provided in an embodiment of this application;

[0074] Figure 5 A schematic diagram of the hardware structure of a chip provided in an embodiment of this application;

[0075] Figure 6 A schematic diagram of a system architecture provided for an embodiment of this application;

[0076] Figure 7 This is a schematic diagram of the structure of Automl provided in the embodiments of this application;

[0077] Figure 8 This is a schematic diagram of the structure of a system for obtaining a neural network model provided in an embodiment of this application;

[0078] Figure 9 A schematic flowchart illustrating a method for obtaining a neural network model provided in an embodiment of this application;

[0079] Figure 10 A schematic block diagram of a hypernetwork model provided for an embodiment of this application;

[0080] Figure 11 A schematic flowchart illustrating a progressive contraction method provided in an embodiment of this application;

[0081] Figure 12 A schematic flowchart illustrating a method for obtaining a neural network model provided in an embodiment of this application;

[0082] Figure 13 A schematic flowchart illustrating the image processing method provided in the embodiments of this application;

[0083] Figure 14 This is a schematic block diagram of the apparatus for obtaining a neural network model provided in the embodiments of this application;

[0084] Figure 15 This is a schematic block diagram of the image processing apparatus provided in the embodiments of this application;

[0085] Figure 16 This is a schematic block diagram of the apparatus for obtaining a neural network model provided in the embodiments of this application;

[0086] Figure 17 This is a schematic block diagram of the image processing apparatus provided in the embodiments of this application. Detailed Implementation

[0087] The technical solutions of the embodiments of this application will be described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0088] Figure 1 A schematic diagram of an artificial intelligence framework is shown, which describes the overall workflow of an artificial intelligence system and is applicable to general artificial intelligence domain needs.

[0089] The above-mentioned artificial intelligence framework will be elaborated in detail from two dimensions: the "intelligent information chain" (horizontal axis) and the "information technology (IT) value chain" (vertical axis).

[0090] The "intelligent information chain" reflects a series of processes from data acquisition to processing. For example, it could be a general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, and intelligent execution and output. In this process, data undergoes a condensation process of "data—information—knowledge—wisdom."

[0091] The "IT value chain" reflects the value that artificial intelligence brings to the information technology industry, from the underlying infrastructure of human intelligence, information (provided and processed by technology) to the industrial ecosystem of systems.

[0092] (1) Infrastructure:

[0093] Infrastructure provides computing power to support artificial intelligence systems, enables them to communicate with the outside world, and provides support through basic platforms.

[0094] Infrastructure can communicate with the outside world through sensors, and its computing power can be provided by smart chips.

[0095] The intelligent chips here can be hardware acceleration chips such as central processing units (CPUs), neural network processing units (NPUs), graphics processing units (GPUs), application-specific integrated circuits (ASICs), and field-programmable gate arrays (FPGAs).

[0096] The basic platform of the infrastructure can include distributed computing frameworks and related platform guarantees and support, such as cloud storage and computing, and interconnected networks.

[0097] For example, for infrastructure, data can be acquired through sensors and external communication, and then this data can be provided to intelligent chips in the distributed computing system provided by the basic platform for computation.

[0098] (2) Data:

[0099] The data at the next layer of infrastructure is used to represent data sources in the field of artificial intelligence. This data includes graphics, images, voice, text, and IoT data from traditional devices, including business data from existing systems and sensor data such as force, displacement, liquid level, temperature, and humidity.

[0100] (3) Data processing:

[0101] The aforementioned data processing typically includes data training, machine learning, deep learning, search, reasoning, and decision-making.

[0102] Among them, machine learning and deep learning can perform intelligent information modeling, extraction, preprocessing, and training on data, including symbolization and formalization.

[0103] Reasoning refers to the process in which, in a computer or intelligent system, the machine thinks and solves problems by simulating human intelligent reasoning, based on reasoning control strategies and using formalized information. Typical functions include search and matching.

[0104] Decision-making refers to the process of making decisions based on intelligent information after reasoning, and it typically provides functions such as classification, sorting, and prediction.

[0105] (4) General abilities:

[0106] After the data processing mentioned above, the results of the data processing can be used to form some general capabilities, such as algorithms or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc.

[0107] (5) Smart products and industry applications:

[0108] Intelligent products and industry applications refer to products and applications of artificial intelligence systems in various fields. They encapsulate overall artificial intelligence solutions, productize intelligent information decision-making, and realize practical applications. Their application areas mainly include: intelligent manufacturing, intelligent transportation, smart home, intelligent healthcare, intelligent security, autonomous driving, safe city, and intelligent terminals.

[0109] The embodiments of this application can be applied to many fields of artificial intelligence, such as intelligent manufacturing, intelligent transportation, smart home, intelligent healthcare, intelligent security, autonomous driving, and safe city.

[0110] Specifically, the method for obtaining neural network models in the embodiments of this application can be applied to fields that require the use of (deep) neural network models, such as autonomous driving, image classification, image retrieval, image semantic segmentation, image quality enhancement, image super-resolution, and natural language processing.

[0111] Below is a brief introduction to two application scenarios: photo album categorization and safe city initiatives.

[0112] Photo album categories:

[0113] When users store a large number of pictures on their terminal devices (such as mobile phones) or cloud storage, recognizing the images in the album can make it easier for users or the system to classify and manage the album, thus improving the user experience.

[0114] The method for obtaining a neural network model according to embodiments of this application can obtain or optimize a neural network model suitable for album classification. This neural network model can then be used to classify images, thereby tagging images of different categories for easy viewing and searching by users. Furthermore, these image classification tags can be provided to the album management system for categorization management, saving users' management time, improving album management efficiency, and enhancing user experience.

[0115] Attribute recognition in the context of safe city:

[0116] In the context of safe city initiatives, various attribute recognition methods are required, such as pedestrian and cyclist attribute recognition. Deep neural network models play a crucial role in these recognition processes due to their powerful capabilities. By employing the neural network model acquisition method described in this application, a neural network model suitable for attribute recognition in safe city scenarios can be obtained or optimized. This neural network model can then be used to process the input road images, thereby identifying different attribute information within the road images.

[0117] Since the embodiments of this application involve the application of a large number of neural network models, for ease of understanding, the relevant terms and concepts of the neural network models that may be involved in the embodiments of this application will be introduced below.

[0118] (1) Neural Network Model

[0119] A neural network model can be composed of neural units, which can refer to units represented by x. s The arithmetic unit that takes an intercept of 1 as input can output the following:

[0120]

[0121] Where s = 1, 2, ..., n, n is a natural number greater than 1, W s For x s The weights are denoted by b, where b is the bias of the neural unit. f is the activation function of the neural unit, used to introduce nonlinear characteristics into the neural network model to convert the input signal in the neural unit into an output signal. The output signal of this activation function can be used as the input of the next convolutional layer; the activation function can be the sigmoid function. The neural network model is a network formed by connecting multiple of the above individual neural units together; that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field, which can be a region composed of several neural units.

[0122] (2) Deep Neural Network Model

[0123] A deep neural network (DNN), also known as a multi-layer neural network model, can be understood as a neural network model with multiple hidden layers. Based on the position of the layers, the internal neural network model of a DNN can be divided into three categories: input layer, hidden layer, and output layer. Generally, the first layer is the input layer, the last layer is the output layer, and the layers in between are hidden layers. The layers are fully connected, meaning that any neuron in the i-th layer is connected to any neuron in the (i+1)-th layer.

[0124] Although DNNs seem complex, the operation of each layer is actually not complicated. Simply put, it involves the following linear relationship expression: in, It is the input vector. It is the output vector. α is the offset vector, W is the weight matrix (also called coefficients), and α() is the activation function. Each layer is simply an adjustment of the input vector. The output vector is obtained through such a simple operation. Because DNNs have many layers, the coefficients W and the offset vector... The number of these parameters is also relatively large. The definitions of these parameters in DNNs are as follows: Taking the coefficient W as an example: Assuming a three-layer DNN, the linear coefficient from the 4th neuron in the second layer to the 2nd neuron in the third layer is defined as... The superscript 3 represents the layer number where coefficient W is located, while the subscript corresponds to the third layer index 2 of the output and the second layer index 4 of the input.

[0125] In summary, the coefficient from the k-th neuron in layer L-1 to the j-th neuron in layer L is defined as...

[0126] It's important to note that the input layer does not have a W parameter. In deep neural network models, more hidden layers allow the network to better represent complex real-world situations. Theoretically, the more parameters a model has, the higher its complexity and "capacity," meaning it can perform more complex learning tasks. Training a deep neural network model is essentially the process of learning the weight matrix, with the ultimate goal of obtaining the weight matrix of all layers in the trained deep neural network model (a weight matrix formed by the vectors W from many layers).

[0127] (3) Convolutional Neural Network Model

[0128] A convolutional neural network (CNN) is a deep neural network model with convolutional structures. A CNN model includes a feature extractor consisting of convolutional layers and subsampling layers, which can be viewed as a filter. A convolutional layer is a layer of neurons in the CNN model that performs convolutional processing on the input signal. In a convolutional layer of a CNN model, a neuron may only be connected to some of its neighboring neurons. A convolutional layer typically contains several feature planes, each composed of rectangularly arranged neural units. Neural units within the same feature plane share weights, which are called the convolutional kernel. Shared weights can be understood as the way image information is extracted regardless of location. The convolutional kernel can be formalized as a matrix of random size, and during the training process of the CNN model, the kernel can learn appropriate weights. Furthermore, the direct benefit of shared weights is reducing the connections between layers in the CNN model, while also reducing the risk of overfitting.

[0129] (4) Recurrent neural networks (RNNs) are used to process sequential data. In traditional neural network models, the layers from the input layer to the hidden layer and then to the output layer are fully connected, but the nodes within each layer are unconnected. While this common neural network model has solved many difficult problems, it is still powerless against many others. For example, to predict the next word in a sentence, you generally need to use the preceding words because the words in a sentence are not independent. RNNs are called recurrent neural networks because the current output of a sequence is related to the previous output. Specifically, the network memorizes previous information and applies it to the calculation of the current output. That is, the nodes within the hidden layer are no longer unconnected but connected, and the input of the hidden layer includes not only the output of the input layer but also the output of the hidden layer at the previous time step. Theoretically, RNNs can process sequential data of any length. Training an RNN is the same as training a traditional CNN or DNN.

[0130] (5) Loss Function

[0131] In training a deep neural network model, to ensure the model's output closely approximates the desired predicted value, we compare the network's prediction with the target value. Based on the difference, we update the weight vector of each layer (usually pre-configuring parameters before the initial update). For example, if the prediction is too high, the weight vector is adjusted to predict a lower value. This adjustment continues until the model predicts the target value accurately or very closely. Therefore, we need to predefine "how to compare the difference between the predicted and target values," which is the loss function or objective function. These are crucial equations for measuring the difference between the predicted and target values. For example, a higher loss function output indicates a greater difference, making the training of a deep neural network model a process of minimizing this loss.

[0132] (6) Backpropagation algorithm

[0133] Neural network models can employ backpropagation (BP) algorithms during training to correct the parameters, thereby minimizing the reconstruction error loss. Specifically, forward propagation of the input signal to the output generates error loss; this error loss information is then propagated back to update the parameters of the neural network model, leading to convergence of the error loss. The backpropagation algorithm is an error-loss-driven backpropagation process designed to obtain optimal parameters for the neural network model, such as the weight matrix.

[0134] like Figure 2 As shown, this application embodiment provides a system architecture 100. In Figure 2 In this process, the data acquisition device 160 is used to collect training data. For a neural network model used for image classification, the training data may include training images and the classification results corresponding to the training images, wherein the results of the training images may be manually pre-annotated.

[0135] After collecting the training data, the data acquisition device 160 stores the training data in the database 130, and the training device 120 trains the target model / rule 101 based on the training data maintained in the database 130.

[0136] The following describes how the training device 120 obtains the target model / rule 101 based on training data. The training device 120 processes the input original image, compares the output image with the original image, and continues until the difference between the output image and the original image is less than a certain threshold, thus completing the training of the target model / rule 101. In this embodiment, the training device 120 can be used to obtain a pre-trained hypernetwork model, transfer the pre-trained hypernetwork model based on the target dataset, and search for sub-network models in the transferred hypernetwork model to obtain the target model / rule 101. The target dataset can be stored in the database 130. In some possible implementations, the training device 120 can also be used to pre-train the hypernetwork model. This hypernetwork model is trained based on the source dataset. This source dataset can also be stored in the database 130.

[0137] The aforementioned target model / rule 101 can be used to implement the image processing method of this application embodiment. Specifically, the target model / rule 101 in this application embodiment can be a neural network model. It should be noted that in practical applications, the training data maintained in the database 130 may not all come from the data acquisition device 160; it may also be received from other devices, such as the target dataset input by the client device 140. Furthermore, it should be noted that the training device 120 may not necessarily train the target model / rule 101 entirely based on the training data maintained in the database 130; it may also obtain training data from the cloud or other sources for model training. The above description should not be construed as limiting the embodiments of this application.

[0138] The target model / rule 101 trained using training device 120 can be applied to different systems or devices, such as... Figure 2 The execution device 110 shown can be a terminal, such as a mobile phone terminal, tablet computer, laptop computer, augmented reality (AR) / virtual reality (VR) device, vehicle terminal, etc., or it can be a server or cloud service. Figure 2 In this embodiment, the execution device 110 is configured with an input / output (I / O) interface 112 for data interaction with external devices. Users can input data to the I / O interface 112 through the client device 140. The input data may include the image to be processed input by the client device.

[0139] The preprocessing module 113 is used to preprocess the input data (such as the image to be processed) received by the I / O interface 112. In this embodiment, the preprocessing module 113 may be omitted, and the calculation module 111 may be used directly to process the input data.

[0140] During the preprocessing of input data by the execution device 110, or during the calculation module 111 of the execution device 110 performing calculations and other related processes, the execution device 110 can call data, code, etc. in the data storage system 150 for corresponding processing, or store the data, instructions, etc. obtained from the corresponding processing into the data storage system 150.

[0141] Finally, I / O interface 112 returns the processing results, such as the image classification results obtained above, to client device 140, thereby providing them to the user.

[0142] It is worth noting that the training device 120 can generate corresponding target models / rules 101 based on different training data for different objectives or tasks. The corresponding target models / rules 101 can be used to achieve the above objectives or complete the above tasks, thereby providing the user with the required results.

[0143] exist Figure 2 In the scenario shown, the user can manually provide input data, which can be done through the interface provided by I / O interface 112. Alternatively, the client device 140 can automatically send input data to I / O interface 112. If user authorization is required for the client device 140 to automatically send input data, the user can set the corresponding permissions in the client device 140. The user can view the output results of the execution device 110 on the client device 140, which can be presented in various forms such as display, sound, or animation. The client device 140 can also act as a data acquisition terminal, collecting the input data and output results of the input I / O interface 112 as new sample data and storing them in the database 130. Alternatively, data can be collected directly from the I / O interface 112 without going through the client device 140, using the input data and output results of the input I / O interface 112 as new sample data and storing them in the database 130.

[0144] It is worth noting that, Figure 2 This is merely a schematic diagram of a system architecture provided in an embodiment of this application. The positional relationships between the devices, components, modules, etc., shown in the diagram do not constitute any limitation. For example, in Figure 2 In this context, the data storage system 150 is an external memory relative to the execution device 110. In other cases, the data storage system 150 may also be placed within the execution device 110.

[0145] like Figure 2As shown, the target model / rule 101 is obtained by training according to the training device 120. The target model / rule 101 can be the neural network model in the embodiments of this application. Specifically, the neural network model constructed in the embodiments of this application can include CNN, deep convolutional neural network (DCNN), recurrent neural network (RNNS), etc.

[0146] Since CNN is a very common neural network model, the following will combine... Figure 3 This section focuses on a detailed explanation of the structure of CNNs. As mentioned in the basic concept introduction above, a convolutional neural network (CNN) model is a deep neural network model with a convolutional structure. It is a deep learning architecture, which refers to an algorithm that updates a neural network model by performing multi-level learning at different abstraction levels. As a deep learning architecture, CNN is a feed-forward artificial neural network model, in which each neuron can respond to the input image.

[0147] The structure of the neural network model specifically used in the image processing method of this application embodiment can be as follows: Figure 3 As shown. In Figure 3 In this model, the Convolutional Neural Network (CNN) 200 may include an input layer 210, convolutional / pooling layers 220 (where the pooling layers are optional), and a neural network model layer 230. The input layer 210 acquires the image to be processed and passes it to the convolutional / pooling layers 220 and the subsequent neural network model layer 230 for processing to obtain the image processing result. The following section... Figure 3 This section provides a detailed introduction to the internal layer structure of CNN 200.

[0148] Convolutional / pooling layers 220:

[0149] Convolutional layers:

[0150] like Figure 3 The convolutional / pooling layer 220 shown may include layers as in Examples 221-226. For instance, in one implementation, layer 221 is a convolutional layer, layer 222 is a pooling layer, layer 223 is a convolutional layer, layer 224 is a pooling layer, layer 225 is a convolutional layer, and layer 226 is a pooling layer; in another implementation, layers 221 and 222 are convolutional layers, layer 223 is a pooling layer, layers 224 and 225 are convolutional layers, and layer 226 is a pooling layer. That is, the output of the convolutional layer can be used as the input to a subsequent pooling layer, or as the input to another convolutional layer to continue the convolution operation.

[0151] The following section will use convolutional layer 221 as an example to introduce the internal working principle of a convolutional layer.

[0152] Convolutional layer 221 can include multiple convolution operators, also known as kernels. In image processing, a convolution operator acts as a filter to extract specific information from the input image matrix. Essentially, a convolution operator can be a weight matrix, which is usually predefined. During the convolution operation, the weight matrix typically processes the input image pixel by pixel (or two pixels by two pixels, depending on the stride) along the horizontal direction, thus extracting specific features from the image. The size of the weight matrix should be related to the image size. It's important to note that the depth dimension of the weight matrix is ​​the same as the depth dimension of the input image; during convolution, the weight matrix extends to the entire depth of the input image. Therefore, convolution with a single weight matrix produces a single-depth convolutional output. However, in most cases, a single weight matrix is ​​not used; instead, multiple weight matrices of the same size (rows × columns) are applied—multiple identical matrices. The outputs of each weight matrix are stacked to form the depth dimension of the convolutional image; this dimension can be understood as being determined by the "multiple" mentioned above. Different weight matrices can be used to extract different features from an image. For example, one weight matrix can be used to extract edge information, another to extract specific colors, and yet another to blur unwanted noise. These multiple weight matrices have the same size (rows × columns), and the resulting convolutional feature maps are also of the same size. These extracted convolutional feature maps are then merged to form the output of the convolution operation.

[0153] The weight values ​​in these weight matrices need to be obtained through extensive training in practical applications. The weight matrices formed by the weight values ​​obtained through training can be used to extract information from the input image, thereby enabling the convolutional neural network model 200 to make correct predictions.

[0154] When a convolutional neural network model 200 has multiple convolutional layers, the earlier convolutional layers (e.g., 221) tend to extract more general features, which can also be called low-level features. As the depth of the convolutional neural network model 200 increases, the features extracted by the later convolutional layers (e.g., 226) become more and more complex, such as high-level semantic features. Features with higher semantic levels are more suitable for the problem to be solved.

[0155] Pooling layer:

[0156] Because it is often necessary to reduce the number of training parameters, pooling layers are often introduced periodically after convolutional layers, such as... Figure 3 Layers 221-226 in example 220 can be a convolutional layer followed by a pooling layer, or multiple convolutional layers followed by one or more pooling layers. In image processing, the sole purpose of pooling layers is to reduce the spatial size of the image. Pooling layers can include average pooling and / or max pooling operators to sample the input image to obtain a smaller image size. The average pooling operator calculates the average value of pixel values ​​within a specific range as the result of average pooling. The max pooling operator takes the pixel with the largest value within a specific range as the result of max pooling. Furthermore, just as the size of the weight matrix in a convolutional layer should be related to the image size, the operators in a pooling layer should also be related to the image size. The size of the output image after processing by the pooling layer can be smaller than the size of the input image of the pooling layer. Each pixel in the output image of the pooling layer represents the average or maximum value of the corresponding sub-region of the input image of the pooling layer.

[0157] Neural network model layer 230:

[0158] After processing by the convolutional / pooling layers 220, the convolutional neural network model 200 is still insufficient to output the required information. As mentioned earlier, the convolutional / pooling layers 220 only extract features and reduce the parameters introduced by the input image. However, to generate the final output information (the required class information or other relevant information), the convolutional neural network model 200 needs to utilize the neural network model layer 230 to generate one or a set of required class numbers of output. Therefore, the neural network model layer 230 may include multiple hidden layers (such as...). Figure 3 As shown in layers 231, 232 to 23n) and output layer 240, the parameters contained in these multi-layer hidden layers can be pre-trained based on relevant training data for specific task types, such as image recognition, image classification, image super-resolution reconstruction, etc.

[0159] After the multiple hidden layers in neural network model layer 230, which is the final layer of the entire convolutional neural network model 200, is the output layer 240. This output layer 240 has a loss function similar to classification cross-entropy, specifically used to calculate the prediction error. Once the entire convolutional neural network model 200 has undergone forward propagation (such as...), the loss function is applied. Figure 3 Propagation from 210 to 240 degrees is considered forward propagation, while backward propagation (e.g.) is completed. Figure 3The propagation from 240 to 210 (backpropagation) will begin to update the weight values ​​and biases of the layers mentioned above, in order to reduce the loss of the convolutional neural network model 200 and the error between the output of the convolutional neural network model 200 through the output layer and the ideal result.

[0160] The structure of the neural network model specifically used in the image processing method of this application embodiment can be as follows: Figure 4 As shown. In Figure 4 In this model, the convolutional neural network (CNN) 200 may include an input layer 210, convolutional / pooling layers 220 (where pooling layers are optional), and neural network model layers 230. Figure 3 compared to, Figure 4 Multiple convolutional / pooling layers in the convolutional / pooling layer 220 are executed in parallel, and the extracted features are input into the neural network model layer 230 for processing.

[0161] It should be noted that, Figure 3 and Figure 4 The convolutional neural network models shown are merely examples of two possible convolutional neural network models for the image processing method of this application embodiment. In specific applications, the neural network model used in the image processing method of this application embodiment may also exist in the form of other network models.

[0162] Furthermore, the neural network model obtained by the method for obtaining a neural network model in the embodiments of this application can be used in the image processing method in the embodiments of this application.

[0163] Figure 5 The present application provides a hardware structure for a chip, which includes a neural network model processor 50. This chip can be configured as follows: Figure 2 The execution device 110 shown is used to perform the calculations of the calculation module 111. This chip can also be located in, for example... Figure 2 The training device 120 shown is used to complete the training work of the training device 120 and output the target model / rule 101. For example... Figure 3 and Figure 4 The algorithms for each layer in the convolutional neural network model shown can all be implemented in, for example... Figure 5 This is achieved in the chip shown.

[0164] The Neural Network Model Processor (NPU) 50 is mounted as a coprocessor on the main central processing unit (CPU) (host CPU), and tasks are assigned by the host CPU. The core of the NPU is the arithmetic circuit 503, and the controller 504 controls the arithmetic circuit 503 to retrieve data from the memory (weight memory or input memory) and perform calculations.

[0165] In some implementations, the arithmetic circuit 503 internally includes multiple process engines (PEs). In some implementations, the arithmetic circuit 503 is a two-dimensional pulsating array. The arithmetic circuit 503 can also be a one-dimensional pulsating array or other electronic circuitry capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit 503 is a general-purpose matrix processor.

[0166] For example, suppose we have an input matrix A, a weight matrix B, and an output matrix C. The arithmetic circuit retrieves the corresponding data of matrix B from the weight memory 502 and caches it in each PE of the arithmetic circuit. The arithmetic circuit retrieves the data of matrix A from the input memory 501 and performs matrix operations with matrix B. The partial result or the final result of the obtained matrix is ​​stored in the accumulator 508.

[0167] The vector computation unit 507 can further process the output of the arithmetic circuit, such as vector multiplication, vector addition, exponentiation, logarithmic operations, size comparisons, etc. For example, the vector computation unit 507 can be used for network computation in non-convolutional / non-FC layers of a neural network model, such as pooling, batch normalization, and local response normalization.

[0168] In some implementations, the vector computation unit 507 can store the processed output vector into a unified buffer 506. For example, the vector computation unit 507 can apply a nonlinear function to the output of the arithmetic circuit 503, such as a vector of accumulated values, to generate activation values. In some implementations, the vector computation unit 507 generates normalized values, merged values, or both. In some implementations, the processed output vector can be used as activation input to the arithmetic circuit 503, for example, for use in subsequent layers of a neural network model.

[0169] The unified memory 506 is used to store input data and output data.

[0170] The direct memory access controller (DMAC) 505 moves input data from external memory to input memory 501 and / or unified memory 506, stores weight data from external memory into weight memory 502, and stores data from unified memory 506 into external memory.

[0171] The bus interface unit (BIU) 510 is used to enable interaction between the main CPU, DMAC and instruction fetch memory 509 via a bus.

[0172] The instruction fetch buffer 509, which is connected to the controller 504, is used to store the instructions used by the controller 504.

[0173] The controller 504 is used to call the instructions cached in the instruction memory 509 to control the operation of the computing accelerator.

[0174] Generally, the unified memory 506, input memory 501, weight memory 502, and instruction fetch memory 509 are all on-chip memories, while the external memory is memory outside the NPU. This external memory can be double data rate synchronous dynamic random access memory (DDR SDRAM), high bandwidth memory (HBM), or other readable and writable memory.

[0175] Among them, neural network models, for example Figure 3 and Figure 4 The operations of each layer in the convolutional neural network model shown can be performed by the computation circuit 503 or the vector computation unit 507. Exemplarily, the pre-training operation of the hypernetwork model in this embodiment can be performed by the computation circuit 503 or the vector computation unit 507. Exemplarily, the transfer operation of the pre-trained neural network model based on the target dataset in this embodiment can be performed by the computation circuit 503 or the vector computation unit 507. Exemplarily, the operations of each layer in the target neural network model in this embodiment can be performed by the computation circuit 503 or the vector computation unit 507.

[0176] The above-mentioned Figure 2 The execution device 110 in the embodiment is capable of executing each step of the image processing method of this application. Figure 5 The chip shown can also be used to perform various steps of the image processing method of the embodiments of this application.

[0177] The above-mentioned Figure 2 The training device 110 in the present application is capable of executing the various steps of the method for obtaining a neural network model according to the embodiments of the present application. Figure 5 The chip shown can also be used to perform the various steps of obtaining the neural network model in the embodiments of this application.

[0178] like Figure 6 As shown, this application embodiment provides a system architecture 300. The system architecture includes a local device 301, a local device 302, an execution device 310, and a data storage system 350, wherein the local devices 301 and 302 are connected to the execution device 310 through a communication network.

[0179] The execution device 310 can be implemented by one or more servers. Optionally, the execution device 310 can be used in conjunction with other computing devices, such as data storage devices, routers, load balancers, etc. The execution device 310 can be deployed on a single physical site or distributed across multiple physical sites. The execution device 310 can use data in the data storage system 350 or call program code in the data storage system 350 to implement the neural network model acquisition method or image processing method of the embodiments of this application.

[0180] Specifically, in one implementation, the execution device 310 can perform the following process:

[0181] Obtain a pre-trained hypernetwork model, which is trained based on the source dataset;

[0182] Obtain the target dataset;

[0183] Transfer learning is performed on the pre-trained hypernetwork model based on the target dataset to obtain the hypernetwork model after transfer learning.

[0184] The target neural network model is obtained by searching for sub-network models in the supernetwork model after transfer learning.

[0185] The device 110 can acquire a target neural network model through the above process, which can be used for image classification or image processing, etc.

[0186] Users can interact with execution device 310 by operating their respective user devices (e.g., local device 301 and local device 302). Each local device can represent any computing device, such as a personal computer, computer workstation, smartphone, tablet, smart camera, smart car or other type of cellular phone, media consumption device, wearable device, set-top box, game console, etc.

[0187] Each user's local device can interact with the execution device 310 through a communication network of any communication mechanism / standard. The communication network can be a wide area network, a local area network, a point-to-point connection, or any combination thereof.

[0188] In one implementation, local devices 301 and 302 obtain relevant parameters of the target neural network model from execution device 310, deploy the target neural network model on local devices 301 and 302, and use the target neural network model for image classification or image processing, etc.

[0189] In another implementation, the target neural network model can be directly deployed on the execution device 310. The execution device 310 obtains the image to be processed from the local devices 301 and 302, and uses the target neural network model to classify or perform other types of image processing on the image to be processed.

[0190] The aforementioned execution device 310 can also be a cloud device, in which case the execution device 310 can be deployed in the cloud; or, the aforementioned execution device 310 can also be a terminal device, in which case the execution device 310 can be deployed on the user terminal side. This application embodiment does not limit this.

[0191] In automachine learning (AutoML) cloud service platforms, users can customize neural network models according to their own needs and tasks. The cloud platform, based on automachine learning, can design and search for networks based on user-defined constraints, and then provide the trained network models to the user. These constraints can include the type of network model, its accuracy, latency, and the platform on which the model runs.

[0192] The method for obtaining a neural network model provided in this application embodiment can obtain a neural network model according to the user's needs, thereby improving the performance of the obtained neural network model and increasing the processing efficiency in the process of obtaining the neural network model.

[0193] Figure 7 A schematic structural diagram of the AutoML framework is shown. (For example...) Figure 7 As shown, AutoML typically predefines a search space, which refers to the searchable range. AutoML continuously generates sub-network model configurations within the search space, forming a closed loop of evaluation-feedback-generation of sub-network model configurations until a superior neural network model is finally found.

[0194] Specifically, the search space is determined based on the specific AutoML task. For example, when the specific task is a neural network model, the search space can include multiple neural network model structural units, and the final neural network model is formed by combining these neural network model units in the search space.

[0195] The controller 710 is used to select different configurations within the search space and assign them to the evaluator 720 for evaluation. Then, it updates the policy, or configuration, based on the evaluation results quantified by the evaluator 720. For example, the controller 710 can select neural network model structural units within the search space, or search for neural network model structural units, and combine them to obtain one or more sub-network models. It then selects one sub-network model from the combined sub-network models and assigns its configuration to the evaluator 720 for evaluation.

[0196] The evaluator 720 evaluates the performance metrics of different configurations and feeds the evaluation results back to the controller 710. For example, the evaluator 720 can evaluate the performance metrics of the sub-network model selected by the controller 710. Performance metrics may include the accuracy and latency of the neural network model. The evaluation results are fed back to the optimizer, which uses the controller 710 to update the configuration until the target neural network model is obtained.

[0197] However, the training cost of AutoML is far higher than that of ordinary neural network models. For example, the training time and computational cost of AutoML are at least an order of magnitude higher than those of ordinary neural network models.

[0198] In addition, AutoML usually requires a large amount of data to obtain excellent neural network models. In some small data scenarios, AutoML often has difficulty training to produce excellent models directly.

[0199] Transfer learning allows knowledge or patterns learned in one domain or task to be applied to different but related domains or problems. Generally, transfer learning enables neural network models to be used with small datasets, reducing training resources. However, transferring neural network models through transfer learning may not meet user needs. For example, it may not meet the user's cost requirements. The neural network model obtained before transfer may not meet the user's cost requirements, while the cost of the transferred neural network model is basically the same as the original model. If the original neural network model could not meet the user's cost requirements, then the transferred neural network model will naturally also not meet the user's cost requirements. Another example is the inability to meet the user's accuracy requirements. The neural network model before transfer is trained on the source dataset, and the architecture of the neural network model obtained after transfer remains basically unchanged. If this architecture is inherently flawed or unsuitable for the target task, then simply fine-tuning the parameters based on the target dataset will not significantly improve the performance of the neural network model; that is, the accuracy of the neural network model with this architecture may not meet the user's accuracy requirements.

[0200] Therefore, effectively reducing the training resources required by AutoML and enabling AutoML to work on small datasets is a major challenge that AutoML needs to overcome in its practical application.

[0201] This application provides a method for obtaining a neural network model based on AutoML, which can be executed by the system for obtaining the neural network. Figure 8 This is a schematic block diagram of a system 800 for obtaining a neural network model according to an embodiment of this application.

[0202] To better understand the execution process of the method for obtaining a neural network model in the embodiments of this application, the following is a description of... Figure 8 The functions of each module in the document will be briefly described.

[0203] The system 800 for acquiring neural network models can be a cloud service device or a mobile terminal. For example, it can be a device with sufficient computing power, such as a computer or server, to acquire neural network models, or it can be a system composed of cloud service devices and mobile terminals.

[0204] The system 800 for acquiring neural network models mainly includes: a pre-training module 810, an input module 820, a transfer module 830, a search module 840, a testing module 850, and an output module 860.

[0205] The pre-training module 810 can be used to pre-train the hypernetwork model and obtain the weights of the hypernetwork model.

[0206] In this context, a hypernetwork model refers to a model that can encompass all subnetwork models in the search space. The weights of the hypernetwork model are the weights of all subnetwork models. In other words, the weights of the subnetwork models can be obtained from the hypernetwork model.

[0207] The hypernetwork model can be a predefined AutoML hypernetwork model. For example, the hypernetwork model can be defined according to the task that the neural network model needs to perform.

[0208] It should be noted that the pre-training module 810 is an optional module, and the pre-training process can be performed by other devices. In this case, the transfer module 830 can receive the hypernetwork model pre-trained by other devices.

[0209] Optionally, the pre-training can be offline training, meaning the pre-training process can be completed offline. It should be understood that online and offline in this embodiment can be different stages relative to the user. Alternatively, it can be understood that the system 800 in the offline stage is unaffected by the user, and the hypernetwork model obtained through offline training can be stored for later processing in the online stage. The system 800 in the online stage can accept user input and perform corresponding operations based on the user's input. For example, the pre-training process can be completed offline; when the user uses the system 800 to obtain the required neural network model, they can directly obtain the pre-trained hypernetwork model through the transfer module 830 without needing to perform the pre-training operation online.

[0210] For example, the pre-training module 810 can be located on a cloud server or on a local device.

[0211] The pre-training module 810 pre-trains the hypernetwork model based on the source dataset.

[0212] The source dataset can be a dataset related to the task that the target neural network model needs to perform. For example, when the target neural network model is used for image classification, the source dataset can include source sample images and their classification labels. For instance, the source dataset could be the publicly available ImageNet dataset.

[0213] The input module 820 can be used to receive user input data. For example, the input module 820 can receive any one or more of the following: target dataset, hyperparameters, target cost, target accuracy, target search time, or target loss function, etc.

[0214] This target dataset is used to fine-tune the supernetwork model output by the pre-trained module 810.

[0215] It should be noted that hyperparameters of a neural network model include parameters that remain unchanged during the training process. Hyperparameters are not obtained through training the neural network model; they are typically determined before training.

[0216] For example, the hyperparameters of a neural network model include: the learning rate of the neural network model, the label smoothing coefficient of the neural network model, or the dropout parameter of the neural network model.

[0217] The target overhead refers to the hardware overhead of the target neural network model output by the output module 860 on the target device.

[0218] The target accuracy refers to the inference accuracy of the target neural network model output by the output module 860.

[0219] The target search time refers to the search time for searching sub-network models within the hypernetwork model to obtain the target neural network model.

[0220] This objective loss function is used to fine-tune the supernetwork model output by the pre-trained module 810.

[0221] The transfer module 830 can be used to perform transfer learning on a pre-trained hypernetwork model based on the target dataset. Alternatively, it can be understood as transferring the weights of the hypernetwork model obtained from the pre-training module 810 to the target dataset.

[0222] The migration module 830 can be located on a cloud server or on a local device.

[0223] For example, the transfer module 830 can receive a pre-trained hypernetwork model sent by the pre-training module 810. For instance, if the transfer module 830 and the pre-training module 810 are located on different devices, the pre-trained hypernetwork model can be transmitted between the transfer module 830 and the pre-training module 810 via a communication network.

[0224] Specifically, the transfer module 830 can fine-tune the weights of the hypernetwork model obtained by the pre-training module 810 based on the target dataset. For example, the target dataset can be the dataset input to the input module 820.

[0225] For example, the transfer module 830 can load the weights of the hypernetwork model output by the pre-trained module 810 and fine-tune the hypernetwork model based on the target dataset input by the user.

[0226] For example, the transfer module 830 can load the weights of the supernetwork model output by the pre-trained module 810 and fine-tune the supernetwork model according to the target loss function input by the user.

[0227] The search module 840 can be used to search for sub-network models in the supernetwork model output by the transfer module 830 to obtain the target neural network model.

[0228] The search module 840 can be located on a cloud server or on a local device.

[0229] For example, the performance metrics of the target neural network model can meet the target performance metrics. That is, the search module 840 can search within the supernetwork model for target subnetwork models whose performance metrics meet the target performance metrics, and determine the target neural network model based on the target subnetwork models. The performance metrics of the subnetwork models may include inference accuracy, hardware overhead, or inference time, etc. The target performance metrics may include target accuracy, target overhead, or target inference time, etc.

[0230] The target performance metric can be a default value or can be input via the input module 820. For example, the user can input the desired target performance metric through the input module 820.

[0231] For example, the target subnetwork model can be a subnetwork model whose inference accuracy reaches the target accuracy.

[0232] For example, the target sub-network model can be a sub-network model that achieves the target hardware overhead and inference accuracy. The hardware overhead testing of the sub-network model can be performed by test module 850, which is located on the target device. In other words, the sub-network model is deployed on the target device, and test module 850 tests its hardware overhead.

[0233] For example, the search duration of the search module 840 can meet the target search duration.

[0234] For example, the search module 840 can search for a target sub-network model in the supernetwork model whose performance index meets the target performance index, and whose search time meets the target search time, and determine the target neural network model based on the target sub-network model.

[0235] Test module 850 is used to test the hardware overhead of different sub-network models on the target device. It should be understood that test module 850 is an optional module. Test module 850 is located on the target device.

[0236] The output module 860 is used to output the target neural network model obtained by the search module 840.

[0237] The following is combined with Figure 9 The method 900 for obtaining a neural network model according to an embodiment of this application will be described in detail. Figure 9 The method shown can be performed by a device that acquires a neural network model, for example, by... Figure 2 The training device 120 shown performs the training, or it is performed by... Figure 8 The system 800 shown is executed. The apparatus for acquiring the neural network model can be a cloud service device or a mobile terminal. For example, a computer, server, or other device with sufficient computing power to execute method 900 can also be a system composed of a cloud service device and a mobile terminal. Method 900 includes steps S910 to S940. Steps S910 to S940 are described in detail below.

[0238] S910, a pre-trained hypernetwork model.

[0239] In the embodiments of this application, the pre-trained supernetwork model can also be understood as the weights of the pre-trained supernetwork model.

[0240] For example, step S910 can be performed by Figure 8 The pre-trained module 810 or Figure 2 The training device 120 performs the operation. It should be understood that this is merely illustrative; in this embodiment, step S910 can also be performed by other devices. That is, the pre-trained hypernetwork model obtained in step S930 can be a model trained by other devices.

[0241] In the embodiments of this application, the hypernetwork model can also be referred to as a hypernetwork or hypermodel.

[0242] In this context, a hypernetwork model refers to a model that can encompass all subnetwork models in the search space. The weights of the hypernetwork model are the weights of all subnetwork models. In other words, the weights of the subnetwork models can be obtained from the hypernetwork model.

[0243] A neural network model is composed of multiple layers of operators stacked together. A neural network model can be represented by a directed acyclic graph composed of multiple layers of operators stacked together. In this directed acyclic graph, each layer is a node, and the operator at each node is a single operator.

[0244] Each layer in a hypernetwork model includes multiple operators, meaning each node has multiple candidate operators. Operators between layers are connected via fully connected layers, and each path in a fully connected layer constitutes a subnetwork model. For example, selecting an operator in each layer creates a neural network model composed of selected operators from multiple layers, forming a subnetwork model. Updating the weights in this path updates the weights of the subnetwork model, which in turn updates the weights of the hypernetwork model, thus achieving the effect of training the hypernetwork model. For example... Figure 10 In the hypernetwork model shown, one of the candidate operators in one layer includes operator 411 and operator 412, etc. All candidate operators in this layer can be convolutions, and the number of channels of operator 411 and operator 412 can be different.

[0245] An operator refers to the basic unit of computation in a neural network model. In the embodiments of this application, an operator can also be understood as a structural unit of a neural network model or a "block" in a neural network model. The relationship between the aforementioned hypernetwork model and subnetwork model can also be understood as follows: each layer of the hypernetwork model contains multiple selectable blocks. In each layer, a block is selected and combined to form a subnetwork model.

[0246] For example, operators may include activation operators, feature extraction operators, normalization operators, and overfitting prevention operators. For instance, activation operators may include rectified linear units (ReLU), sigmoid, etc. Feature extraction operators may include convolution, full connection, etc. Normalization operators may include batch normalization, etc. Overfitting prevention operators may include pooling, etc.

[0247] The sub-network models within a hypernetwork model can have the same network topology. Specifically, the direction of data flow between blocks constituting the sub-network model can be the same. The kernel size, number of layers, or number of channels of the sub-network models within the hypernetwork model can differ.

[0248] The hypernetwork model can be predefined. For example, a search space can be predefined based on the task required by the target neural network model. This search space can serve as the searchable range in step S940. Alternatively, the hypernetwork model can be understood as being defined based on the task required by the target neural network model. Specifically, the number or types of selectable operators in the hypernetwork model can be determined based on the task required by the target neural network. For example, the task required by the target neural network may include image classification, image segmentation, or object detection. For example, step S910 can be completed offline. For instance, when a user obtains the required neural network model, they can directly obtain a pre-trained hypernetwork model without needing to perform pre-training operations online.

[0249] For example, step S910 is an optional step. Method 900 can be executed starting from step S920.

[0250] For example, the hypernetwork model can be pre-trained based on the source dataset.

[0251] The source dataset can be a large dataset, which ensures that the hypernetwork model is trained sufficiently and can obtain a hypernetwork model with higher accuracy.

[0252] It should be noted that the source dataset can be a dataset related to the task that the target neural network model needs to perform. The source dataset can include source sample data and the corresponding labels for the source sample data.

[0253] For example, when the target neural network model is used for image classification, the source dataset can include source sample images and their classification labels. For instance, the source dataset could be the publicly available ImageNet dataset.

[0254] For example, when the target neural network model is used for image segmentation, the source dataset can include source sample images and classification labels of pixels in the source sample images.

[0255] For example, when a target neural network model is used for object detection, the source dataset can include source sample images and object classification labels and bounding boxes in the source sample images.

[0256] Optionally, step S910 includes: pre-training the hypernetwork model based on a single-path algorithm.

[0257] Specifically, in each training iteration, only one single-path sub-network model is activated, or it can be understood as selecting only one sub-network model from the super-network model, updating the weights of that sub-network model, and continuing the iteration until training is complete.

[0258] Optionally, the hypernetwork model can be pre-trained based on the progressive shrinking (PS) algorithm. Figure 11 A schematic diagram of a progressive shrinkage algorithm is shown. (For example...) Figure 11 As shown, the maximum subnetwork model is trained first, and then the subnetwork models with variable convolutional kernels, variable number of layers, and variable number of channels are trained step by step.

[0259] Among them, the largest subnetwork model refers to the subnetwork model with the largest convolutional kernel, the largest number of layers (depth), and the largest number of channels (width) in the supernetwork model.

[0260] Specifically, training a sub-network model with variable convolutional kernels can be done by sampling multiple sub-network models from the super-network model that have the largest number of training layers and the largest number of channels. That is, in this training phase, the multiple sub-network models being trained have D layers and W channels, and the convolutional kernel sizes of these multiple sub-network models may be different. Here, D represents the maximum number of layers in the super-network model, and W represents the maximum number of channels in the super-network model.

[0261] For example, training a subnetwork model with variable convolutional kernels can be done by training multiple subnetwork models with the largest number of layers, the largest number of channels, and different convolutional kernel sizes using a random single-path algorithm.

[0262] Specifically, training a sub-network model with a variable number of layers can be done by training multiple sub-network models that sample the largest number of channels from the super-network model. That is, in this training phase, the number of channels in the multiple sub-network models being trained is W, and the kernel sizes and the number of layers in these multiple sub-network models may differ. Here, W represents the maximum number of channels in the super-network model.

[0263] For example, a sub-network model with a variable number of training layers can be trained by sampling multiple sub-network models with different numbers of layers and different convolutional kernels using a random single-path sampling method. Alternatively, a sub-network model with a variable number of training channels can be trained by randomly sampling different sub-network models.

[0264] Specifically, training a subnetwork model with a variable number of channels can be done by sampling multiple subnetwork models from a supernetwork model. That is, during this training phase, the kernel sizes of the multiple subnetwork models being trained may be different, the number of layers of the multiple subnetwork models may be different, and the number of channels of the multiple subnetwork models may be different.

[0265] For example, a sub-network model with a variable number of training channels can be trained by sampling multiple different sub-network models using a random single-path method.

[0266] Specifically, subnetwork models with variable convolutional kernels, variable number of layers, or variable number of channels can be trained using knowledge distillation.

[0267] Knowledge distillation refers to transferring knowledge from one neural network model to another. The knowledge in a neural network model can be understood as a mapping between its inputs and outputs. This mapping is determined by the parameters of the neural network model. In other words, knowledge distillation can be understood as transferring the parameters of one neural network model to another.

[0268] Specifically, knowledge distillation refers to training a student network model using the output of a pre-trained teacher network model and the real labels of the training samples. In the embodiments of this application, the teacher network model refers to the maximum sub-network model, and the student network model refers to a sub-network model with variable convolutional kernels, variable number of layers, or variable number of channels.

[0269] For example, training a subnetwork model with variable convolutional kernels through knowledge distillation means inputting the source data into the trained maximum subnetwork model, obtaining the output value of the maximum subnetwork model, and then training a subnetwork model with variable convolutional kernels based on the output value and the labels corresponding to the source sample data.

[0270] Because sub-network models share weights, interference may occur between different sub-network models during supernetwork training. The progressive shrinking algorithm reduces this mutual influence during training, allowing the resulting supernetwork model to support various architectures. These architectures can include sub-network models with different numbers of layers, channels, and kernel sizes. After the supernetwork model is trained, suitable sub-network models can be selected without additional training of the searched sub-network models, or in other words, sub-network models do not need to be retrained, while still ensuring that the accuracy of the sub-network models meets the pre-training requirements. Furthermore, during the training of the supernetwork model, it is not necessary to train each sub-network model independently; the sub-network models within the supernetwork model can achieve accuracy similar to independently trained sub-network models.

[0271] S920, Obtain input data, which includes the target dataset.

[0272] Optionally, the input data may also include any one or more of the following: hyperparameters, target cost, target accuracy, target search time, or target loss function.

[0273] For example, this step can be performed by Figure 8 The input module 820 is executed.

[0274] The target dataset can be determined based on the task that the target neural network model needs to perform. That is, the sub-network models in the pre-trained supernetwork model perform the same task as the target neural network model. Alternatively, it can be understood as the task corresponding to the source dataset being the same as the task corresponding to the target dataset. For example, both may be used for image classification; or both may be used for image segmentation; or both may be used for object detection.

[0275] For example, when a target neural network model is used to perform image classification, the target dataset may include target sample images and their classification labels. For instance, if the target neural network model is used to perform vehicle recognition, the target dataset may include target vehicle images and their classification labels.

[0276] For example, when a target neural network model is used for image segmentation, the target dataset can include target sample images and classification labels for the pixels in the target sample images. Similarly, when a target neural network model is used for object detection, the target dataset can include an object detection dataset. This object detection dataset can include target sample images, object classification labels for the objects in the target sample images, and the bounding boxes of the objects.

[0277] For example, the target dataset can be a dataset input by the user or a dataset obtained from other devices. For instance, if step S920 is performed by a cloud service device, then the other device can be the target device. The target device can be the device where the target neural network model is to be deployed.

[0278] S930: Transfer learning is performed on the pre-trained hypernetwork model based on the target dataset to obtain the hypernetwork model after transfer learning.

[0279] Transfer learning on a pre-trained hypernetwork model based on a target dataset can be used to fine-tune a pre-trained hypernetwork model based on the target dataset.

[0280] Fine-tuning refers to applying a pre-trained model to a target dataset and adapting the model's parameters to the target dataset.

[0281] Transfer learning on a pre-trained hypernetwork model specifically refers to transferring the weights of the pre-trained hypernetwork model.

[0282] Transfer learning transfers knowledge learned on a source dataset to a target dataset. In other words, it transfers the weights of a pre-trained hypernetic network model to the target dataset.

[0283] For example, step S930 can be performed by Figure 8 Migration module 830 or Figure 2 The training device 120 performs the operation. It should be understood that this is only for illustration, and in the embodiments of this application, step S930 can also be performed by other devices.

[0284] Specifically, a pre-trained hypernetwork model is obtained, or the weights of a pre-trained hypernetwork model are loaded, and the hypernetwork model is fine-tuned based on the target dataset. It should be noted that the device performing step S910 and the device performing step S930 can be the same or different. For example, when the device performing step S910 and the device performing step S930 are different devices, the pre-trained hypernetwork model can be transmitted via a communication network.

[0285] Optionally, step S930 includes fine-tuning the pre-trained supernetwork model using a single-path algorithm. This allows for uniform sampling of subnetwork models and their training, improving training effectiveness. Furthermore, it reduces memory usage and enables efficient training.

[0286] Fine-tuning of pre-trained hypernetwork models using a single-path algorithm includes:

[0287] Select a sub-network model from the pre-trained supernetwork model, calculate the weight gradient of the sub-network model based on the target dataset, update the weights of the sub-network model based on the weight gradient of the sub-network model, and obtain the updated supernetwork model; repeat the above steps until the updated supernetwork model meets the termination condition, and obtain the supernetwork model after transfer learning.

[0288] In other words, in each training iteration, only the sub-network model of a single path is activated, or it can be understood as selecting only one sub-network model from the supernetwork model, updating the weights of that sub-network model, and continuing the iteration until training is complete. During each training iteration, only the weights of the selected sub-network model are activated and updated.

[0289] Among them, selecting a sub-network model from the hypernetwork model can be done by randomly selecting a sub-network model from the hypernetwork model.

[0290] The following example illustrates the method for updating the weights of the currently selected sub-network model, using a single iteration of training as an example.

[0291] During an iterative training process, a sub-network model is selected in the supernetwork model during each forward propagation. That is, the target sample data is input into the sub-network model, the loss value corresponding to the output of the sub-network model is calculated through the loss function, and the weight gradient of the current sub-network model is calculated based on the loss value through backpropagation. The weights of the sub-network model are adjusted according to the weight gradient.

[0292] The loss function's value indicates the difference between the classification label of the target sample image and the predicted label output by the sub-network model. The weights of the sub-network model are updated based on this difference until the predicted label of the neural network model and the label of the training data are very close. For example, a higher loss function value indicates a greater difference, so training the neural network model becomes a process of minimizing this function value. In some cases, the loss function can also be the objective function.

[0293] Optionally, step S930 includes: selecting N from the pre-trained supernetwork model. b Each model calculates N based on the target dataset. b The weight gradients of each sub-network model, based on N b The weight gradients of the sub-network models are used to update the weights of the pre-trained supernetwork model, resulting in an updated supernetwork model. The above steps are repeated until the updated supernetwork model meets the termination condition, resulting in a supernetwork model after transfer learning.

[0294] Each time a sub-network model is selected, only the single-path sub-network model is activated, or it can be understood as each time a sub-network model is selected from the super-network model, only one sub-network model is selected.

[0295] The following example, using a single iteration, illustrates the method for updating the weights of a hypernetwork model.

[0296] During each forward propagation, a sub-network model is selected from the supernetwork model; that is, the target sample image is input into the sub-network model, and the loss function is calculated. Based on this loss function value, backpropagation is performed to calculate the weight gradient of the current sub-network model. This process is repeated N times. b This process, which involves selecting N, is called the next step. b Secondary sub-network model, calculating N b The weight gradient of the secondary sub-network model, accumulated N b The weight gradient is obtained once. Then, based on this accumulated weight gradient, the weights of the hypernetwork model are updated once; this process can be considered an iterative process. Iteration continues until the termination condition is met, resulting in the hypernetwork model after transfer learning, i.e., the hypernetwork model transfer is complete. Where N... b N is a positive integer. b It can be pre-set or user-inputted. The value of the loss function indicates the difference between the classification label of the target sample image and the predicted label output by the sub-network model.

[0297] Where, N b The cumulative weight gradient can satisfy:

[0298]

[0299] Where dW represents the weight gradient of the supernetwork model, and L represents the value of the loss function during the i-th forward propagation in one iteration.

[0300] For example, updating the weights of a hypernetwork model can include updating the weights of the hypernetwork model by subtracting the accumulated weight gradients from the current weights of the hypernetwork model.

[0301] Alternatively, updating the supernetwork model weights can include subtracting the product of the accumulated weight gradients and the learning rate from the current supernetwork model weights. For example, the current supernetwork model weights can satisfy:

[0302] W j =W j-1 -lr*dW

[0303] Among them, W j W represents the weights of the hypernetwork model after the j-th iteration. j-1Let represent the weights of the hypernetwork model after the (j-1)th iteration, and lr represent the learning rate.

[0304] Optionally, the termination condition includes a repetition count greater than or equal to the first iteration count. In this case, the iteration count can also be understood as the number of times the weights of the hypernetwork model are updated.

[0305] Optionally, the termination condition includes that the inference accuracy of the updated hypernetwork model is greater than or equal to the first inference accuracy. For example, the inference accuracy of the hypernetwork model can be the inference accuracy of at least one subnetwork model in the hypernetwork model.

[0306] In the embodiments of this application, "reasoning" can also be referred to as "prediction".

[0307] Optionally, the termination condition may include: within a preset time interval, the change in inference accuracy of the z sub-network models is less than a set threshold. These z sub-network models can be pre-specified. That is, z sub-network models can be pre-specified, and the accuracy of these z sub-network models is tested in each iteration. If the change in accuracy of these z sub-network models is not significant within a certain period or a certain number of iterations, the migration can be terminated, meaning the supernetwork model migration is complete.

[0308] Since sub-network models may share weights, updating the weights of the current sub-network model based on the weight gradients calculated in each backpropagation would interfere with other sub-network models that share weights. In the solution of this application embodiment, multiple forward and backward propagations are performed in each iteration, accumulating the weight gradients of multiple sub-network models in one iteration, and updating the weights of the supernetwork model only once. This reduces mutual interference between different sub-network models, improves the accuracy of the supernetwork model, and increases its training speed.

[0309] The loss function in step S930 can be a pre-set loss function or a target loss function input by the user.

[0310] It should be understood that the above fine-tuning methods are only examples. Other methods that can fine-tune the pre-trained hypernetwork model are also applicable to step S930. The embodiments of this application do not limit the methods of fine-tuning the hypernetwork model.

[0311] S940 searches for sub-network models from the supernetwork model after transfer learning to obtain the target neural network model.

[0312] For example, this step can be performed by Figure 8 Search module 840 or Figure 2 The training equipment 120 is used for execution.

[0313] For example, the performance metrics of the target neural network model can meet the target performance metrics. That is, the target sub-network model can be searched within the supernetwork model after transfer learning, and the target neural network model is determined based on the target sub-network model. The target sub-network model can be a sub-network model whose performance metrics meet the target performance metrics. The performance metrics of the sub-network model can include inference accuracy, hardware overhead, or inference time, etc. The target performance metrics can include target accuracy, target overhead, or target inference time, etc.

[0314] The target overhead refers to the hardware overhead of the target neural network model on the target device.

[0315] The target accuracy refers to the inference accuracy of the target neural network model.

[0316] The inference time of this target refers to the inference time of the target neural network model.

[0317] The target performance metric can be a pre-set target performance metric or a target performance metric input by the user. For example, step S920 also includes obtaining the target cost, target accuracy, or target inference time, etc.

[0318] For example, if the target performance metric is target accuracy, then the target sub-network model can be a sub-network model whose inference accuracy reaches the target accuracy.

[0319] For example, if the target performance metrics include target accuracy and target cost, then the target sub-network model can be a sub-network model whose hardware cost reaches the target cost and whose inference accuracy reaches the target accuracy. Exemplarily, the hardware cost testing of the sub-network model can be performed by the test module 850. Specifically, when testing the hardware cost of the sub-network model, the sub-network model can be deployed on the target device to test its hardware cost.

[0320] For example, the search time for the target subnetwork model can meet the target search time.

[0321] The search time for this objective refers to the search time required to find the sub-network model within the supernetwork model.

[0322] For example, in the supernetwork model, a target subnetwork model whose performance index meets the target performance index is searched, and the search time meets the target search time. The target neural network model is then determined based on the target subnetwork model.

[0323] The target search duration can be a preset target search duration or a target search duration entered by the user.

[0324] For example, the target neural network model is obtained by searching for sub-network models in the super-network model after transfer learning using a reinforcement learning algorithm.

[0325] For example, the target neural network model is obtained by searching for sub-network models in the supernetwork model after transfer learning using an evolutionary algorithm.

[0326] Searching for sub-network models in the supernetwork model after transfer learning using evolutionary algorithms can include:

[0327] Step 1: Determine n first sub-network models based on the supernetwork model after transfer learning. These n first sub-network models can be used as the initial population.

[0328] For example, n sub-network models can be extracted from the supernetwork model after transfer learning. These n sub-network models are the n first sub-network models.

[0329] Step 2: Adjust the structure of the n first sub-network models to obtain n second sub-network models.

[0330] For example, adjusting the structure of n first sub-network models can be achieved through operations such as crossover and mutation.

[0331] Step 3: Select n third sub-network models from the n first sub-network models and n second sub-network models, and use these n third sub-network models as the n first sub-network models in Step 2. These n third sub-network models constitute the new population.

[0332] Repeat steps two through three above until n third sub-network models satisfy the search termination condition.

[0333] The target neural network model is determined based on n third sub-network models.

[0334] Where n is a positive integer greater than 1. n can be preset or input by the user. For example, the value of n can be obtained through the input module 820.

[0335] The search termination criteria can be preset or determined based on user input data.

[0336] For example, the search termination condition could be that the number of repetitions is greater than or equal to the number of second iterations. This second number of iterations can be preset or entered by the user.

[0337] For example, the search termination condition can be that the accuracy of at least p third sub-network models out of n third sub-network models meets the target accuracy. Here, p is a positive integer, p ≤ n. p can be pre-set or user-inputted. For example, the value of p can be obtained through input module 820.

[0338] For example, the search termination condition could be that the search duration reaches the target search duration.

[0339] For example, the search termination condition can be that the hardware overhead of at least q out of n third sub-network models satisfies the target overhead. Here, q is a positive integer, q ≤ n. q can be pre-set or user-inputted. For instance, the value of q can be obtained through input module 820.

[0340] In the embodiments of this application, "extraction" can also be understood as "sampling".

[0341] Further, in step one, determining n first sub-network models based on the supernetwork model after transfer learning may include:

[0342] Select n fourth sub-network models from the hypernetwork model after transfer learning;

[0343] Obtain the hardware overhead of the n fourth sub-network models on the target device;

[0344] Based on the hardware overhead, the structure of the n fourth sub-network models is adjusted to obtain n first sub-network models.

[0345] The following example illustrates a method for searching for sub-network models within a supernetwork model after transfer learning to obtain the target neural network model.

[0346] For example, n sub-network models are extracted from the supernetwork model after transfer learning. The structure of these n sub-network models is adjusted according to their hardware overhead on the target device, resulting in n adjusted sub-network models. These n adjusted sub-network models are used as the initial population. Crossover and mutation are used to generate new n sub-network models. N sub-network models are selected from these 2n models to form a new population, and the iteration continues until the search termination condition is met. The final n sub-network models are the search results.

[0347] Where n is a positive integer greater than 1. n can be preset or input by the user. For example, the value of n can be obtained through the input module 820.

[0348] The search termination condition can be preset or determined based on user input. For example, the search termination condition could be that the current iteration number has reached the second iteration number. This second iteration number can be preset or entered by the user.

[0349] For example, the search termination condition can be that at least p of the n sub-network models obtained in the current iteration meet the target accuracy. Here, p is a positive integer, p ≤ n. p can be pre-set or user-inputted. For instance, the value of p can be obtained through input module 820.

[0350] For example, the iteration termination condition could be that the search time reaches the target search time.

[0351] For example, the termination condition for this iteration can be that the hardware cost of at least q of the n sub-network models obtained in the current iteration meets the target cost. Here, q is a positive integer, q ≤ n. q can be pre-set or input by the user. For example, the value of q can be obtained through input module 820.

[0352] It should be understood that the above search termination conditions are merely examples, and search termination conditions can be set as needed. For example, search termination conditions may include the two conditions mentioned above.

[0353] For example, extracting n sub-network models from a supernetwork model may include: randomly selecting n sub-network models from the supernetwork model.

[0354] For example, selecting n sub-network models from 2n sub-network models to form a new population can take many forms. The following example illustrates one way to select n sub-network models from 2n sub-network models.

[0355] For example, selecting n sub-network models from the 2n sub-network models to form a new population can include: randomly selecting n sub-network models from the 2n sub-network models to form a new population.

[0356] Alternatively, selecting n sub-network models from the 2n sub-network models to form a new population may include: testing the hardware overhead of the 2n sub-network models, and selecting n sub-network models to form a new population based on hardware overhead constraints.

[0357] Alternatively, selecting n sub-network models from the 2n sub-network models to form a new population may include: testing the inference accuracy of the 2n sub-network models and selecting the n sub-network models with the highest inference accuracy to form a new population.

[0358] Alternatively, selecting n sub-network models from the 2n sub-network models to form a new population may include: testing the inference accuracy and hardware overhead of the 2n sub-network models, and selecting the n sub-network models with the highest inference accuracy within the hardware overhead limit to form a new population.

[0359] For example, adjusting the sub-network model structure based on the hardware overhead of the n sub-network models on the target device to obtain adjusted n sub-network models includes: adjusting the sub-network model structure according to the probability of sub-network model structure adjustment, wherein the adjusted sub-network model can meet the target overhead. The probability of sub-network model structure adjustment is determined based on the hardware overhead of the sub-network model.

[0360] For example, for a subnetwork model with high hardware overhead, the probability of adjusting the current subnetwork model to a smaller subnetwork model is greater than the probability of adjusting the current subnetwork model to a larger subnetwork model. For a subnetwork model with low hardware overhead, the probability of adjusting the current subnetwork model to a larger subnetwork model is greater than the probability of adjusting the current subnetwork model to a smaller subnetwork model. The hardware overhead can be determined relative to a target overhead. For example, a subnetwork model with an overhead greater than the target overhead can be considered a subnetwork model with high hardware overhead, and a subnetwork model with an overhead less than the target overhead can be considered a subnetwork model with low hardware overhead. Alternatively, the hardware overhead can also be determined relative to other benchmarks, and this embodiment does not limit this.

[0361] For example, the target device may include a GPU or an NPU.

[0362] This heuristic search approach allows us to perceive the hardware overhead of the subnetwork model on the target device and adjust the structure of the subnetwork model based on the hardware overhead, so that the final subnetwork model can meet the target overhead.

[0363] Furthermore, the target subnetwork model is fed back to the user. The target subnetwork model can be a single subnetwork model or multiple subnetwork models. For example, the target subnetwork model includes m subnetwork models, where m is a positive integer, m ≤ n. The n subnetwork models in the search results include these m subnetwork models. m can be pre-set or input by the user.

[0364] Specifically, search results can be presented to users based on their needs. The following example illustrates the specific format for presenting search results to users.

[0365] For example, m sub-network models can be fed back to the user, who can then select the desired sub-network model as the target neural network model. For instance, the m sub-network models fed back to the user could be m sub-network models randomly selected from the search results. Alternatively, the m sub-network models fed back to the user could be the m sub-network models with the highest accuracy in the search results; further, these m sub-network models could be fed back to the user in a ranking based on accuracy. Another example is that the m sub-network models fed back to the user could be the m sub-network models with the lowest cost in the search results; further, these m sub-network models could be fed back to the user in a ranking based on cost. Yet another example is that the m sub-network models fed back to the user could be the m sub-network models with the highest accuracy within a target cost range.

[0366] Furthermore, the accuracy of the m sub-network models can be fed back to the user, who can then select the desired sub-network model as the target neural network model.

[0367] Furthermore, the overhead of the m sub-network models can be fed back to the user, who can then select the desired sub-network model as the target neural network model.

[0368] According to the solution in this application, by transferring the pre-trained hypernetwork model to the target dataset, a high-performance hypernetwork model can be obtained even when the target dataset is small. This empowers applications in small data scenarios and greatly improves the accuracy of AutoML in such scenarios.

[0369] At the same time, to meet different user needs, such as user cost / accuracy requirements, a neural network model that meets the user's needs can be obtained by searching the sub-network models in the hypernetwork model, and adapted to the target dataset to meet the user's needs, such as meeting the user's cost / accuracy requirements.

[0370] Meanwhile, the weights of the hypernetic network model are shared between different datasets. Both the source and target datasets are datasets related to the same task, which enables efficient transfer learning of AutoML. During transfer, only the weights of the hypernetic network model are fine-tuned without adjusting the structure of the hypernetic network model. This can greatly improve the transfer efficiency of AutoML and reduce the training time required by at least one order of magnitude, or even reach the training time of ordinary neural network models.

[0371] Furthermore, the transfer learning time of the hypernetwork model provided in this embodiment is close to that of a regular neural network model. In other words, compared to methods that obtain the target neural network model through transfer learning of a regular neural network model, the method for obtaining the neural network model in this embodiment better meets the user's refined cost / accuracy requirements under the same training time. It achieves a more accurate target neural network model with the same overhead.

[0372] Furthermore, for the same task, such as image classification, when users need multiple neural network models, there is no need to design and train neural network models separately for each deployment scheme or user needs. Only one supernetwork model needs to be trained. The weights of the supernetwork model are shared or transferred to different datasets, resulting in neural network models that meet the different overhead / accuracy requirements of users, which greatly reduces training costs.

[0373] Furthermore, the hypernetwork model trained using the progressive shrinkage method can support various architectural settings. After the hypernetwork model is trained, a suitable subnetwork model can be selected from it without additional training. For example, in the embodiments of this application, during the process of searching for the target neural network model from the hypernetwork model after transfer learning, it is not necessary to retrain the subnetwork model. In other words, the subnetwork model does not need to be retrained, and the accuracy of the subnetwork model can still meet the requirements of pre-training.

[0374] The following is through Figure 12 This paper illustrates the process of deploying multiple neural network models using the method described in the embodiments of this application. Figure 12 A schematic flowchart illustrating a method for obtaining a neural network model according to an embodiment of this application is shown. Figure 12 The method includes steps S1110 to S1140. Figure 12 The method can be regarded as Figure 9 One embodiment of method 900 is described above. The specific implementation can be found in the aforementioned method 900. To avoid unnecessary repetition, the following description... Figure 12 When using this method, appropriate omissions of repetitive descriptions are acceptable.

[0375] The following will be Figure 12 The method is explained in two stages: offline and online.

[0376] Offline phase:

[0377] S1110, a pre-trained supernetwork model.

[0378] Specifically, a predefined search space, or a predefined hypernetwork model, is used. The hypernetwork model is pre-trained based on a source dataset. This source dataset can be a dataset related to the task that the target neural network model needs to perform.

[0379] For example, when the target neural network model is used for image classification, the source dataset can include source sample images and their classification labels. Figure 12 As shown, the source dataset can be the publicly available ImageNet dataset.

[0380] Online phase:

[0381] S1120, Obtain the pre-trained supernetwork model.

[0382] For example, pre-trained hypernetwork models can be loaded.

[0383] S1130, Obtain the target dataset.

[0384] The target dataset can be determined based on the tasks that the target neural network model needs to perform.

[0385] For example, when the target neural network model is used for image classification, the target dataset may include target sample images and classification labels for the target sample images.

[0386] For example, the target dataset can be a target dataset input by the user.

[0387] S1140 performs transfer learning on a pre-trained hypernetwork model based on the target dataset.

[0388] It should be noted that, Figure 12 Only one target dataset is shown in this embodiment; the number of target datasets is not limited in this application. If the user inputs multiple target datasets, the pre-trained hypernetwork model can be transferred to different target datasets.

[0389] For example, multiple target datasets include bird datasets or vehicle datasets. Step S1140 includes performing transfer learning on a pre-trained hypernetwork model based on the bird dataset to obtain a transfer-learned hypernetwork model 1, and performing transfer learning on a pre-trained hypernetwork model based on the vehicle dataset to obtain a transfer-learned hypernetwork model 2.

[0390] S1150, search within the supernetwork model after transfer learning to obtain the target neural network model.

[0391] Specifically, a search can be performed on the hypernetwork model after transfer learning based on different user needs to obtain a target neural network model that meets the different user needs.

[0392] For example, such as Figure 12 As shown, different user requirements may include achieving target accuracy 1 for the inference accuracy of target neural network 1, and achieving target accuracy 2 for the inference accuracy of target neural network 2.

[0393] It should be noted that, Figure 12 This application uses only two target accuracies as examples of two user requirements. The number of user requirements and the specific content of user requirements are not limited in this embodiment.

[0394] In this way, for the same task, such as image classification, when a user needs multiple neural network models, it is not necessary to design and train neural network models separately for each deployment scheme or user requirement. Only the hypernetwork model needs to be trained once. The pre-trained hypernetwork model is then transferred to the target dataset, and a search is performed based on different user needs to obtain neural network models that meet the different overhead / accuracy requirements of users, which greatly reduces training costs.

[0395] It should be understood that the above explanation uses the application of a target neural network model to image classification as an example only. The method for obtaining a neural network model provided in this application can be applied to other computer vision tasks, such as object detection and image segmentation.

[0396] For example, the target neural network model can also be applied to non-visual tasks, such as natural language processing or speech recognition.

[0397] In different application scenarios, the source dataset and the target dataset can be determined according to the application scenario.

[0398] For example, when a target neural network model is applied to speech recognition, the source dataset may include source sample audio signals and corresponding classification labels, while the target dataset may include target sample audio signals and corresponding classification labels.

[0399] Figure 13 A schematic flowchart of an image processing method 1200 provided in an embodiment of this application is shown. The method can be executed by a device or equipment capable of image processing, such as a terminal device, computer, server, etc.

[0400] Figure 13 The target neural network model used in the image processing method 1200 can be obtained through the above... Figure 9 The method or Figure 12 The method described in the previous section is used to construct the method. Method 1200 includes steps S1210 to S1220. The specific implementation of method 1200 can be referred to the aforementioned method 900. To avoid unnecessary repetition, repeated descriptions will be appropriately omitted when introducing method 1200 below.

[0401] S1210, Obtain the image to be processed.

[0402] The image to be processed may be an image captured by a camera of a terminal device (or a computer, server, or other device or equipment), or it may be an image obtained from within the terminal device (or a computer, server, or other device or equipment) (e.g., an image stored in the terminal device's photo album, or an image obtained by the terminal device from the cloud). This application embodiment does not limit this.

[0403] S1220: The target neural network model is used to process the image to be processed, and the processing result of the image to be processed is obtained.

[0404] The target neural network model is obtained by searching for sub-network models within the hypernetwork model. The hypernetwork model is obtained through transfer learning of a pre-trained hypernetwork model based on the target dataset. The pre-trained hypernetwork model is trained on the source dataset. Both the source and target datasets are datasets related to image processing tasks.

[0405] Optionally, a target neural network model can be used to classify the image to be processed and output the classification result.

[0406] The target neural network model is obtained by searching for sub-network models within the supernetwork model. The supernetwork model is obtained through transfer learning of a pre-trained supernetwork model based on the target dataset. The pre-trained supernetwork model is trained on the source dataset. The target dataset includes target sample images and their classification labels, while the source dataset includes source sample images and their classification labels.

[0407] For detailed steps on obtaining the neural network model, please refer to the aforementioned method 900, which will not be repeated here.

[0408] It should be understood that the above explanation uses the application of a target neural network model to image classification as an example only. The method for obtaining a neural network model provided in this application can be applied to other computer vision tasks, such as object detection and image segmentation.

[0409] For example, the target neural network model can also be applied to non-visual tasks, such as natural language processing or speech recognition.

[0410] In different application scenarios, the source dataset and the target dataset can be determined according to the application scenario.

[0411] For example, when a target neural network model is applied to speech recognition, the source dataset may include source sample audio signals and corresponding classification labels, while the target dataset may include target sample audio signals and corresponding classification labels.

[0412] The following will combine Figures 14 to 17 This document describes in detail the device embodiments of this application. It should be understood that the device in the embodiments of this application can execute the methods described in the foregoing embodiments of this application. That is, the specific working processes of the various products described below can be referred to the corresponding processes in the foregoing method embodiments.

[0413] Figure 14 This is a schematic block diagram of the apparatus 1300 for acquiring a neural network model provided in an embodiment of this application. It should be understood that the apparatus 1300 can perform... Figure 9 or Figure 12 A method for obtaining a neural network model. For example, device 1300 can be used to... Figure 1 Training equipment 120, or Figure 6 The execution device 310 in the middle, or Figure 8 The system 800 in the device 1300 includes an acquisition unit 1310 and a processing unit 1320.

[0414] The acquisition unit 1310 acquires a pre-trained hypernetwork model, which is trained based on the source dataset; acquires a target dataset, the task corresponding to the target dataset being the same as the task corresponding to the source dataset; the processing unit 1320 transfers the pre-trained hypernetwork model based on the target dataset to obtain a hypernetwork model after transfer learning; and searches for subnetwork models in the hypernetwork model after transfer learning to obtain a target neural network model.

[0415] Optionally, as an example, the pre-trained supernetwork model is trained using a progressive shrinkage method.

[0416] Optionally, as an embodiment, the processing unit 1320 is specifically configured to: select a sub-network model from the pre-trained supernetwork model, calculate the weight gradient of the sub-network model based on the target dataset, update the weights of the sub-network model based on the weight gradient of the sub-network model to obtain an updated sub-network model, and obtain an updated supernetwork model based on the updated sub-network model; repeat the above steps until the updated supernetwork model meets the termination condition to obtain the supernetwork model after transfer learning; wherein, the termination condition includes at least one of the following: the number of repetitions is greater than or equal to the number of first iterations; the inference accuracy of the updated supernetwork model is greater than or equal to the first inference accuracy.

[0417] Optionally, as an embodiment, the processing unit 1320 is specifically configured to: select N from the pre-trained hypernetwork model b Each model calculates the N based on the target dataset. b The weight gradients of each sub-network model, based on the N... bThe weight gradient update of the N sub-network model b The weights of each sub-network model are used to obtain the updated N. b A sub-network model, based on the updated N b The updated hypernetwork model is obtained from the N subnetwork models. b The value is a positive integer; repeat the above steps until the updated hypernetwork model meets the termination condition to obtain the hypernetwork model after transfer learning, wherein the termination condition includes at least one of the following: the number of repetitions is greater than or equal to the number of the first iteration; the inference accuracy of the updated hypernetwork model is greater than or equal to the first inference accuracy.

[0418] Optionally, as an embodiment, the processing unit 1320 is specifically used for: Step 1: determining n first sub-network models based on the supernetwork model after transfer learning, where n is an integer greater than 1; Step 2: adjusting the structure of the n first sub-network models to obtain n second sub-network models; Step 3: selecting n third sub-network models from the n first sub-network models and the n second sub-network models, and using the n third sub-network models as the n first sub-network models in Step 2; repeating Step 2 to Step 3 until the n third sub-network models meet the search termination condition, the search termination condition including at least one of the following: the number of repetitions is greater than or equal to the number of second iterations, or the inference accuracy of at least p third sub-network models among the n third sub-network models is greater than or equal to the target accuracy; determining the target neural network model based on the n third sub-network models.

[0419] Optionally, as an embodiment, the processing unit 1320 is specifically used to: select n fourth sub-network models from the supernetwork model after transfer learning; obtain the hardware overhead of the n fourth sub-network models on the target device; and adjust the structure of the n fourth sub-network models based on the hardware overhead to obtain the n first sub-network models.

[0420] Figure 15 This is a schematic block diagram of the image processing apparatus 1400 provided in an embodiment of this application. It should be understood that apparatus 1300 can perform... Figure 13 Image processing methods. For example, device 1400 can be... Figure 1 The execution device 110 in the middle, or Figure 6 The local device 301 or execution device 310 in the device 1400. The device 1400 includes: an acquisition unit 1410 and a processing unit 1420.

[0421] The acquisition unit 1410 is used to acquire the image to be processed; the processing unit 1420 is used to perform image processing on the image to be processed using a target neural network model and output the processing result; wherein the target neural network model is obtained by searching for a sub-network model in a hypernetwork model, the hypernetwork model is obtained by transfer learning on a pre-trained hypernetwork model based on the target dataset, the pre-trained hypernetwork model is trained based on the source dataset, and the task corresponding to the target dataset is the same as the task corresponding to the source dataset.

[0422] Optionally, as an example, the pre-trained supernetwork model is trained using a progressive shrinkage method.

[0423] Optionally, as an embodiment, the hypernetwork model is obtained by transfer learning from a pre-trained hypernetwork model based on a target dataset, including: selecting a sub-network model from the pre-trained hypernetwork model, calculating the weight gradient of the sub-network model based on the target dataset, updating the weights of the sub-network model based on the weight gradient of the sub-network model to obtain an updated sub-network model, and obtaining an updated hypernetwork model based on the updated sub-network model; repeating the above steps until the updated hypernetwork model meets the termination condition; wherein, the termination condition includes at least one of the following: the number of repetitions is greater than or equal to the number of first iterations; the inference accuracy of the updated hypernetwork model is greater than or equal to the first inference accuracy.

[0424] Optionally, as an embodiment, the hypernetwork model is obtained by transfer learning from a pre-trained hypernetwork model based on a target dataset, including: the hypernetwork model is obtained by selecting N from the pre-trained hypernetwork model. b Each model calculates the N based on the target dataset. b The weight gradients of each sub-network model, based on the N... b The weight gradient update of the N sub-network model b The weights of each sub-network model are used to obtain the updated N. b A sub-network model, based on the updated N b The updated hypernetwork model is obtained from the N subnetwork models. b The value is a positive integer; repeat the above steps until the updated hypernetwork model satisfies the termination condition, wherein the termination condition includes at least one of the following: the number of repetitions is greater than or equal to the number of the first iteration; the inference accuracy of the updated hypernetwork model is greater than or equal to the first inference accuracy.

[0425] Optionally, as an embodiment, the target neural network model is obtained by searching for sub-network models in a supernetwork model, including: determining n first sub-network models based on the supernetwork model, where n is an integer greater than 1; adjusting the structure of the n first sub-network models to obtain n second sub-network models; selecting n third sub-network models from the n first sub-network models and the n second sub-network models, and updating the n third sub-network models to the n first sub-network models; repeating the above steps until the n third sub-network models satisfy the search termination condition; determined based on the n third sub-network models; wherein, the search termination condition includes at least one of the following: the number of repetitions is greater than or equal to the number of second iterations, or, the inference accuracy of at least p third sub-network models among the n third sub-network models is greater than or equal to the target accuracy.

[0426] Optionally, as an embodiment, determining n first sub-network models based on the supernetwork model includes: selecting n fourth sub-network models from the supernetwork model; obtaining the hardware overhead of the n fourth sub-network models on the target device; and adjusting the structure of the n fourth sub-network models based on the hardware overhead to obtain the n first sub-network models.

[0427] It should be noted that the aforementioned devices 1300 and 1400 are embodied in the form of functional units. The term "unit" here can be implemented in software and / or hardware, without specific limitations.

[0428] For example, a "unit" can be a software program, a hardware circuit, or a combination of both that implements the above functions. The hardware circuit may include an application-specific integrated circuit (ASIC), electronic circuitry, a processor (e.g., a shared processor, a proprietary processor, or a group processor) and memory for executing one or more software or firmware programs, integrated logic circuitry, and / or other suitable components that support the described functions.

[0429] Therefore, the units of the various examples described in the embodiments of this application can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0430] Figure 16 This is a schematic diagram of the hardware structure of the device for obtaining a neural network model provided in an embodiment of this application. Figure 16The apparatus 3000 for acquiring a neural network model (specifically, this apparatus 3000 can be a computer device) includes a memory 3001, a processor 3002, a communication interface 3003, and a bus 3004. The memory 3001, processor 3002, and communication interface 3003 are interconnected via the bus 3004. For example, the apparatus 3000 could be... Figure 1 Training equipment 120, or Figure 6 The execution device 310 in the middle, or Figure 8 System 800 in the middle.

[0431] The memory 3001 may be a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 3001 may store a program. When the program stored in the memory 3001 is executed by the processor 3002, the processor 3002 performs the various steps of the method for obtaining a neural network model according to the embodiments of this application. Exemplarily, the processor 3002 may perform the steps described above... Figure 9 Steps S920 to S940 in the method shown Figure 12 Steps S1120 to S1150 in the method shown.

[0432] The processor 3002 may be a general-purpose central processing unit (CPU), microprocessor, application-specific integrated circuit (ASIC), graphics processing unit (GPU), or one or more integrated circuits, used to execute relevant programs to implement the method of obtaining a neural network model according to the method embodiment of this application.

[0433] The processor 3002 can also be an integrated circuit chip with signal processing capabilities; for example, it could be... Figure 5 The chip shown is described. In implementation, each step of the method for obtaining a neural network model according to this application can be completed through integrated logic circuits in the hardware of the processor 3002 or through software instructions.

[0434] The processor 3002 described above can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules can be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. This storage medium is located in memory 3001. The processor 3002 reads the information in memory 3001 and, in conjunction with its hardware, completes the functions required by the units included in the apparatus for obtaining a neural network model in the embodiments of this application, or executes the method for obtaining a neural network model in the embodiments of this application.

[0435] The communication interface 3003 uses transceiver devices, such as, but not limited to, transceivers, to enable communication between the device 3000 and other devices or communication networks. For example, pre-trained hypernetwork models or target datasets can be obtained through the communication interface 3003.

[0436] Bus 3004 may include a pathway for transmitting information between various components of device 3000 (e.g., memory 3001, processor 3002, communication interface 3003).

[0437] Figure 17 This is a schematic diagram of the hardware structure of the image processing apparatus according to an embodiment of this application. Figure 17 The image processing apparatus 4000 shown includes a memory 4001, a processor 4002, a communication interface 4003, and a bus 4004. The memory 4001, processor 4002, and communication interface 4003 are interconnected via the bus 4004. For example, the apparatus 4000 can be... Figure 1 The execution device 110 in the middle, or Figure 6 Local device 301 or execution device 310.

[0438] The memory 4001 can be a ROM, static storage device, or RAM. The memory 4001 can store programs, and when the program stored in the memory 4001 is executed by the processor 4002, the processor 4002 and the communication interface 4003 are used to execute various steps of the image processing method of this application embodiment. Specifically, the processor 4002 can execute the steps described above... Figure 13 Steps S1210 to S1220 in the method shown.

[0439] The processor 4002 may be a general-purpose CPU, microprocessor, ASIC, GPU or one or more integrated circuits, used to execute relevant programs to achieve the functions required by the units in the image processing apparatus of this application embodiment, or to execute the image processing method of this application method embodiment.

[0440] The processor 4002 can also be an integrated circuit chip with signal processing capabilities; for example, it could be... Figure 5 The chip shown. In implementation, each step of the image processing method of this application embodiment can be completed by the integrated logic circuit of the hardware in the processor 4002 or by instructions in the form of software.

[0441] The processor 4002 described above can also be a general-purpose processor, DSP, ASIC, FPGA, or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules can be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. This storage medium is located in memory 4001. The processor 4002 reads the information in memory 4001 and, in conjunction with its hardware, completes the functions required by the units included in the image processing apparatus of this application embodiment, or executes the image processing method of the method embodiment of this application.

[0442] The communication interface 4003 uses a transceiver device, such as, but not limited to, a transceiver, to enable communication between the device 4000 and other devices or communication networks. For example, an image to be processed can be acquired through the communication interface 4003.

[0443] Bus 4004 may include a pathway for transmitting information between various components of device 4000 (e.g., memory 4001, processor 4002, communication interface 4003).

[0444] It should be noted that although only the memory, processor, and communication interface are shown in the above-described devices 3000 and 4000, those skilled in the art should understand that in specific implementations, devices 3000 and 4000 may also include other devices necessary for normal operation. Furthermore, depending on specific needs, those skilled in the art should understand that devices 3000 and 4000 may also include hardware devices for implementing other additional functions. In addition, those skilled in the art should understand that devices 3000 and 4000 may only include the devices necessary for implementing the embodiments of this application, and may not necessarily include... Figure 16 and Figure 17 All the devices shown.

[0445] It should be understood that the processor in the embodiments of this application can be a central processing unit (CPU), but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0446] It should also be understood that the memory in the embodiments of this application can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDR SDRAM), enhanced synchronous DRAM (ESDRAM), synchronous linked DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0447] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.

[0448] It should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. A and B can be singular or plural. Additionally, the character " / " in this article generally indicates an "or" relationship between the preceding and following related objects, but it can also represent an "and / or" relationship. Please refer to the context for a more accurate understanding.

[0449] In this application, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or multiple items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0450] It should be understood that in the various embodiments of this application, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0451] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0452] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0453] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0454] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0455] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0456] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0457] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An image processing method, characterized in that, include: Obtain the image to be processed; The target neural network model is used to perform image processing on the image to be processed, and the processing result is output. The target neural network model is obtained by searching for sub-network models in the hypernetwork model. The hypernetwork model is obtained by transfer learning of the pre-trained hypernetwork model based on the target dataset. The pre-trained hypernetwork model is trained based on the source dataset. The task corresponding to the target dataset is the same as the task corresponding to the source dataset.

2. The method according to claim 1, characterized in that, The pre-trained hypernetwork model was obtained through progressive shrinkage training.

3. The method according to claim 1 or 2, characterized in that, The hypernetwork model is obtained by transfer learning from a pre-trained hypernetwork model based on the target dataset, including: The supernetwork model is obtained by selecting a subnetwork model from the pre-trained supernetwork model, calculating the weight gradient of the subnetwork model based on the target dataset, updating the weights of the subnetwork model based on the weight gradient of the subnetwork model, obtaining an updated subnetwork model, and obtaining an updated supernetwork model based on the updated subnetwork model; repeating the above steps until the updated supernetwork model meets the termination condition. The termination condition includes at least one of the following: the number of repetitions is greater than or equal to the number of the first iterations; the inference accuracy of the updated hypernetwork model is greater than or equal to the first inference accuracy.

4. The method according to claim 1 or 2, characterized in that, The hypernetwork model is obtained by transfer learning from a pre-trained hypernetwork model based on the target dataset, including: The hypernetwork model is obtained by selecting N from the pre-trained hypernetwork model. b Each sub-network model calculates the N based on the target dataset. b The weight gradients of each sub-network model, based on the N... b The weight gradient update of the N sub-network model b The weights of each sub-network model are used to obtain the updated N. b A sub-network model, based on the updated N b The updated hypernetwork model is obtained from the N subnetwork models. b The value is a positive integer; repeat the above steps until the updated hypernetwork model satisfies the termination condition, wherein the termination condition includes at least one of the following: the number of repetitions is greater than or equal to the number of the first iteration; the inference accuracy of the updated hypernetwork model is greater than or equal to the first inference accuracy.

5. The method according to claim 1 or 2, characterized in that, The target neural network model is obtained by searching for sub-network models within a supernetwork model, including: The target neural network model is determined by: identifying n first sub-network models (n being an integer greater than 1) based on the supernetwork model; adjusting the structure of the n first sub-network models to obtain n second sub-network models; selecting n third sub-network models from the n first and n second sub-network models; updating the n third sub-network models to the n first sub-network models; repeating the above steps until the n third sub-network models meet the search termination condition; and determining the target neural network model based on the n third sub-network models. The search termination condition includes at least one of the following: the number of repetitions is greater than or equal to the number of second iterations, or the inference accuracy of at least p of the n third sub-network models is greater than or equal to the target accuracy.

6. The method according to claim 5, characterized in that, The step of determining n first sub-network models based on the supernetwork model includes: selecting n fourth sub-network models from the supernetwork model; obtaining the hardware overhead of the n fourth sub-network models on the target device; and adjusting the structure of the n fourth sub-network models based on the hardware overhead to obtain the n first sub-network models.

7. An image processing apparatus, characterized in that, include: The acquisition unit is used to acquire the image to be processed; Processing unit, used for: The target neural network model is used to perform image processing on the image to be processed, and the processing result is output. The target neural network model is obtained by searching for sub-network models in the hypernetwork model. The hypernetwork model is obtained by transfer learning of the pre-trained hypernetwork model based on the target dataset. The pre-trained hypernetwork model is trained based on the source dataset. The task corresponding to the target dataset is the same as the task corresponding to the source dataset.

8. The apparatus according to claim 7, characterized in that, The pre-trained hypernetwork model was obtained through progressive shrinkage training.

9. The apparatus according to claim 7 or 8, characterized in that, The hypernetwork model is obtained by transfer learning from a pre-trained hypernetwork model based on the target dataset, including: The supernetwork model is obtained by selecting a subnetwork model from the pre-trained supernetwork model, calculating the weight gradient of the subnetwork model based on the target dataset, updating the weights of the subnetwork model based on the weight gradient of the subnetwork model, obtaining an updated subnetwork model, and obtaining an updated supernetwork model based on the updated subnetwork model; repeating the above steps until the updated supernetwork model meets the termination condition. The termination condition includes at least one of the following: the number of repetitions is greater than or equal to the number of the first iterations; the inference accuracy of the updated hypernetwork model is greater than or equal to the first inference accuracy.

10. The apparatus according to claim 7 or 8, characterized in that, The hypernetwork model is obtained by transfer learning from a pre-trained hypernetwork model based on the target dataset, including: The hypernetwork model is obtained by selecting N from the pre-trained hypernetwork model. b Each sub-network model calculates the N based on the target dataset. b The weight gradients of each sub-network model, based on the N... b The weight gradient update of the N sub-network model b The weights of each sub-network model are used to obtain the updated N. b A sub-network model, based on the updated N b The updated hypernetwork model is obtained from the N subnetwork models. b The value is a positive integer; repeat the above steps until the updated hypernetwork model satisfies the termination condition, wherein the termination condition includes at least one of the following: the number of repetitions is greater than or equal to the number of the first iteration; the inference accuracy of the updated hypernetwork model is greater than or equal to the first inference accuracy.

11. The apparatus according to claim 7 or 8, characterized in that, The target neural network model is obtained by searching for sub-network models within a supernetwork model, including: The target neural network model is determined by: identifying n first sub-network models (n being an integer greater than 1) based on the supernetwork model; adjusting the structure of the n first sub-network models to obtain n second sub-network models; selecting n third sub-network models from the n first and n second sub-network models; updating the n third sub-network models to the n first sub-network models; repeating the above steps until the n third sub-network models meet the search termination condition; and determining the target neural network model based on the n third sub-network models. The search termination condition includes at least one of the following: the number of repetitions is greater than or equal to the number of second iterations, or the inference accuracy of at least p of the n third sub-network models is greater than or equal to the target accuracy.

12. The apparatus according to claim 11, characterized in that, The step of determining n first sub-network models based on the hypernetwork model includes: selecting n fourth sub-network models from the hypernetwork model; Obtain the hardware overhead of the n fourth sub-network models on the target device; adjust the structure of the n fourth sub-network models based on the hardware overhead to obtain the n first sub-network models.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores program code for execution by the device, the program code including methods for performing any one of claims 1 to 6.

14. A chip, characterized in that, The chip includes a processor and a data interface. The processor reads instructions stored in the memory through the data interface to execute the method as described in any one of claims 1 to 6.

Images