Image processing methods, apparatus, electronic devices and storage media

Abstract

This application discloses an image processing method, apparatus, electronic device, and storage medium. The method includes: processing an image to be processed using a target neural network to obtain an image processing result; wherein the target neural network is constructed using target network modules searched from a first operator search space and a second operator search space; the first operator search space includes multiple different first network modules, each designed based on the computing performance of a first computing platform; the second operator search space includes multiple different second network modules, each designed based on the computing performance of a second computing platform; each network module includes a combination of one or more operators. Implementing this application embodiment enables image processing using neural networks suitable for different computing platforms, ensuring compatibility with the computing capabilities of different platforms and achieving high image processing accuracy on various computing platforms.

Description

Technical Field

[0001] This application relates to the field of machine learning technology, specifically to image processing methods, apparatus, electronic devices, and storage media. Background Technology

[0002] With the development of machine learning technology, machine learning algorithms based on artificial neural networks are widely used in image processing tasks. Among these, neural network design is a crucial aspect of machine learning, as a well-designed network structure typically leads to significant performance improvements. However, current methods primarily rely on manual design of neural networks, heavily depending on the design experience of researchers, making it challenging to design high-performance neural networks. Furthermore, neural networks often require substantial computational resources for image processing tasks, while smartphones and other electronic devices have limited computing resources. This means that manually designed neural networks may not function effectively on these resource-constrained devices, potentially leading to a decrease in image processing accuracy. Summary of the Invention

[0003] This application discloses an image processing method, apparatus, electronic device, and storage medium, which can perform image processing through neural networks suitable for different computing platforms, so as to be compatible with the computing power of different computing platforms and achieve high image processing accuracy on different computing platforms.

[0004] This application discloses an image processing method, comprising: processing an image to be processed using a target neural network to obtain an image processing result; wherein the target neural network is constructed using target network modules searched from a first operator search space and a second operator search space; the first operator search space includes multiple different first network modules, the first network modules being designed based on the computing performance of a first computing platform; the second operator search space includes multiple different second network modules, the second network modules being designed based on the computing performance of a second computing platform; each network module includes a combination of one or more operators.

[0005] This application discloses an image processing apparatus, comprising: a processing module, configured to process an image to be processed using a target neural network to obtain an image processing result; wherein the target neural network is constructed using target network modules searched from a first operator search space and a second operator search space; the first operator search space includes multiple different first network modules, each designed based on the computing performance of a first computing platform; the second operator search space includes multiple different second network modules, each designed based on the computing performance of a second computing platform; each network module includes a combination of one or more operators.

[0006] This application discloses an electronic device, including a memory and a processor. The memory stores a computer program, and when the computer program is executed by the processor, the processor implements any of the image processing methods disclosed in this application.

[0007] This application discloses a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements any of the image processing methods disclosed in this application.

[0008] Compared with related technologies, the embodiments of this application have the following beneficial effects:

[0009] Different operator search spaces are designed for different computing platforms, and target network modules are searched from these spaces to construct a target neural network. This results in a target neural network that is applicable to both the first and second computing platforms. Therefore, when using this target neural network for image processing, it is compatible with the computing capabilities of both platforms, achieving high image processing accuracy on both. Furthermore, the search for target network modules, the construction of candidate subnetworks, and the selection of the target neural network can all be automated by electronic devices. This frees researchers from the task of designing neural network structures, making neural network design more efficient. It may even uncover network structures that are difficult for humans to imagine, further improving network performance and image processing accuracy. Attached Figure Description

[0010] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 This is a flowchart illustrating the network architecture search method in related technologies;

[0012] Figure 2 This is a schematic diagram of the method flow of a neural network search method disclosed in one embodiment;

[0013] Figure 3A This is an example diagram of a first operator search space disclosed in one embodiment;

[0014] Figure 3B This is an example diagram of a second operator search space disclosed in one embodiment;

[0015] Figure 4This is an example diagram of a candidate subnetwork disclosed in one embodiment;

[0016] Figure 5 This is a structural example diagram of a U-shaped network framework disclosed in one embodiment;

[0017] Figure 6 This is a schematic diagram of the method flow of another neural network search method disclosed in one embodiment;

[0018] Figure 7 This is a schematic flowchart of a method for pruning the optimal candidate subnetwork, as disclosed in an embodiment.

[0019] Figure 8 This is a schematic flowchart of an image processing method disclosed in one embodiment;

[0020] Figure 9 This is a schematic diagram of the structure of a neural network search device disclosed in one embodiment;

[0021] Figure 10 This is a schematic diagram of the structure of an image processing apparatus disclosed in one embodiment;

[0022] Figure 11 This is a schematic diagram of the structure of an electronic device disclosed in one embodiment. Detailed Implementation

[0023] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0024] It should be noted that the terms "comprising" and "having," and any variations thereof, in the embodiments and accompanying drawings of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the steps or units listed, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or devices.

[0025] This application discloses an image processing method, apparatus, electronic device, and storage medium, which can perform image processing through neural networks suitable for different computing platforms, so as to be compatible with the computing power of different computing platforms and achieve high image processing accuracy on different computing platforms.

[0026] The image processing method disclosed in this application is based on a non-manually constructed neural network. To better illustrate the image processing method disclosed in this application, the Network Architecture Search (NAS) method in related technologies is first introduced. The NAS method is a method for constructing neural networks without manual intervention. Please refer to... Figure 1 , Figure 1 This is a flowchart illustrating the network architecture search method in related technologies. For example... Figure 1 As shown, the NAS method first requires designing a search space 110.

[0027] Assuming the set of all potential neural network structures in the world constitutes the network structure space, the search space 110 can be viewed as a subset of the network structure space. Researchers or developers can artificially define a search space 110 within the aforementioned network structure space based on prior knowledge. The search space 110 may include multiple network modules, and each network module may include a combination of various operators. The network module defines the type, number, and connection relationships of the operators within the module.

[0028] After defining the search space 110, the NAS network 120 can be searched from the search space 110 based on a certain search strategy. The search strategy refers to how to select from the search space. The search strategy can be to identify good network structures from the search space 110, thereby constructing the NAS network 120.

[0029] The NAS network 120 can be further evaluated on the target dataset to verify the performance of the network results. The aforementioned search strategy aims to find some networks with the highest performance (such as accuracy) in the search space 110. Therefore, the results of the performance evaluation can be fed back into the search strategy to guide the next round of search until a high-performing neural network is finally found.

[0030] Based on this, the following content will introduce the image processing method disclosed in the embodiments of this application.

[0031] The image processing method disclosed in this application can be applied to electronic devices such as smartphones and tablets. The image processing method may include the following steps:

[0032] Electronic devices utilize a target neural network constructed based on an operator search space to process images and obtain image processing results. The image to be processed can be an image captured by the imaging device of the electronic device (such as a camera), or it can be transmitted to the electronic device from other devices; there are no specific limitations.

[0033] The image processing operations that the target neural network performs on the image to be processed can be determined based on the data type of the training dataset during the training of the target neural network. The image processing result is determined by the type of image processing operations that the target neural network can perform.

[0034] The target neural network is not manually constructed; it can be constructed by an electronic device using target network modules searched from a first operator search space and a second operator search space. The first operator search space includes multiple different first network modules, which are designed based on the computing performance of a first computing platform. The second operator search space includes multiple different second network modules, which are designed based on the computing performance of a second computing platform. Each network module includes a combination of one or more operators.

[0035] It should be noted that the electronic device that constructs the target neural network and the electronic device that applies the target neural network for image processing can be different electronic devices. For example, the target neural network can be constructed using electronic devices with strong computing power, such as cloud servers or personal computers, and then image processing can be performed using electronic devices with weaker computing power, such as smartphones or tablets. The aforementioned first and second computing platforms can be two different computing platforms with weaker computing power, and there is no specific limitation.

[0036] The following section provides a detailed description of how to construct the aforementioned target neural network. For ease of understanding, the following section focuses on the neural network search methods applicable to constructing the target neural network. In other words, the target neural network in the image processing method disclosed in this application can be constructed using any of the neural network search methods disclosed in the following embodiments.

[0037] Please see Figure 2 , Figure 2 This is a flowchart illustrating a neural network search method disclosed in one embodiment. This method can be applied to electronic devices such as cloud servers and personal computers, and is not specifically limited thereto. Figure 2 As shown, the image processing method may include the following steps:

[0038] 210. Construct multiple candidate subnetworks using the target network modules searched from the first operator search space and the second operator search space.

[0039] The first operator search space and the second operator search space can be custom search spaces, designed based on the computing performance of different computing platforms.

[0040] A computing platform can refer to any processor or system-on-a-chip (SoC) with computing capabilities, without any specific limitation. For example, a computing platform can be an embedded neural network processing unit (NPU); or, a computing platform can also be a computing system in which a central processing unit (CPU), a graphics processing unit (GPU), and a digital signal processing unit (DSP) work together.

[0041] The first and second computing platforms can be two different computing platforms with different computing performance. Different computing performance means that the same operator may achieve different accuracies or different computing efficiencies on different computing platforms.

[0042] For example, the first and second computing platforms can be NPUs from different chip vendors. The first computing platform is more efficient and accurate at running depthwise separable convolutions, while the second computing platform is more efficient and accurate at running dilated convolutions. Alternatively, the first computing platform is more efficient at performing addition and concatenation (CAT) operations than the second computing platform. Or, the first computing platform is more efficient at running convolutional networks with 16 channels, while the second computing platform is more efficient at running convolutional networks with 32 channels.

[0043] Therefore, the first operator search space can be designed based on the computational performance of the first computing platform, and the second operator search space can be designed based on the computational performance of the second computing platform. The first operator search space may include multiple different first network modules, and the second operator search space may include multiple different second network modules. The first operator search space includes at least one first network module that is different from the second network module. Each network module may include a combination of one or more operators, so different network modules differ in at least one of the following: the number of operators, the connection relationship between operators, or the type of operators.

[0044] Optionally, the first network module included in the first operator search space has higher computational efficiency on the first computing platform; while the second network module included in the second operator search space has higher computational efficiency on the second computing platform.

[0045] In other words, the first network module included in the first operator search space has a higher computational efficiency on the first computing platform than on the second computing platform; the second network module included in the second operator search space has a higher computational efficiency on the second platform than on the first computing platform, thus enabling the final searched target neural network to be compatible with the computational performance of two different computing platforms and to have high computational efficiency on different computing platforms.

[0046] For example, please refer to Figure 3A , Figure 3A This is an example diagram of the search space for a first operator disclosed in one embodiment. For example... Figure 3A As shown:

[0047] The first network module 311 may include: a convolutional neural network (CNN), a depth-separable convolution operator, and a concatenation (CAT) operator.

[0048] The first network module 312 may include: three CNN networks, a depthwise separable convolution operator, and an addition operator.

[0049] The first network module 313 may include: two CNN networks and an addition operator.

[0050] The first network module 314 may include: three CNN networks and an addition operator.

[0051] like Figure 3A As shown, the first network module 311 and the first network module 313 include the same number of operators and the same connection relationships between the operators, but the types of operators are different. The first network module 311 includes depthwise separable convolution operators, while the first network module 313 does not.

[0052] Please see Figure 3B , Figure 3B This is an example diagram of a second operator search space disclosed in one embodiment. For example... Figure 3B As shown:

[0053] The second network module 323 may include: two CNN networks and a splicing operator.

[0054] The second network module 324 may include: three CNN networks and a splicing operator.

[0055] The second network module 321 may include: a CNN network, a dilated convolution operator, and a splicing operator.

[0056] The second network module 322 may include: a CNN network, two dilated convolution operators, and a splicing operator.

[0057] like Figure 3B As shown, the second network module 321 and the second network module 323 include the same number of operators and the same connection relationships between the operators, but the types of operators are different. The second network module 321 includes a dilated convolution operator, while the second network module 323 does not.

[0058] In addition, from Figure 3A and Figure 3B As can be seen, the first computing platform is more efficient at running depthwise separable convolutions, while the second computing platform is more efficient at running dilated convolutions. Therefore, the first network module tends to use depthwise separable convolution operators to perform convolution operations, while the second network module tends to use dilated convolution operators.

[0059] Because the second computing platform has higher efficiency in concatenation operations but lower efficiency in addition operations, the second network module tends to use concatenation operators rather than addition operators to perform feature fusion. The first computing platform, however, can achieve high computational efficiency in both addition and concatenation operations; therefore, the first network module can use either addition or concatenation operators to perform feature fusion.

[0060] In other words, the first network module included in the first operator search space and the second network module included in the second operator search space can perform the same type of operation through different operators. The first network module is more efficient on the first computing platform, and the second network module is more efficient on the second computing platform.

[0061] It is evident that the network modules included in the first and second operator search spaces are applicable to different computing platforms. After designing the first and second operator search spaces, target network modules for constructing candidate subnetworks can be searched from them based on search strategies.

[0062] Search strategies may include, but are not limited to: random search, gradient-based search, reinforcement learning-based search, and evolutionary algorithm-based search.

[0063] Each candidate subnetwork may include one or more target network modules. Some or all of the network modules included in the candidate subnetwork can be searched from the first operator search space and the second operator search space. That is, each candidate subnetwork is constructed using the target network modules searched from the first operator search space and the second operator search space.

[0064] It should be noted that the hyperparameters of neural networks are mainly divided into two types: one is the hyperparameters related to the network structure, and the other is the hyperparameters related to model training.

[0065] Hyperparameters related to network structure may include: the number of network layers, the type of network layers, the number of neurons in each layer, or the activation function, etc.

[0066] Hyperparameters related to model training may include: loss function, optimization method, batch size, number of iterations, learning rate, regularization method and coefficients, initialization method, etc.

[0067] Therefore, step 210 described above can be seen as determining hyperparameters related to the network structure, including determining the number, type, and connection relationships of network layers. In the embodiments of this application, a network layer may include one or more network modules, and the multiple network modules included in each network layer may be independent of each other.

[0068] For an example, please refer to 4. Figure 4 This is an example diagram of a candidate subnetwork disclosed in one embodiment. For example... Figure 4 As shown, candidate subnetwork 410 and candidate subnetwork 420 include the same number and type of network layers, namely network layer 1, network layer 2, network layer 3, network layer 4, and network layer 5. The candidate subnetwork 410 and candidate subnetwork 420 differ in the connection relationships between their network layers.

[0069] Candidate subnetworks 410 and 430 differ in the number of network layers, the type of network layers, and the connection relationships between network layers.

[0070] Figure 4 The candidate subnetworks 410, 420 and 430 shown are three different candidate subnetworks.

[0071] 220. Determine the target neural network based on multiple candidate subnetworks.

[0072] In this embodiment, after each candidate subnetwork is constructed, it can be trained and its performance verified to determine whether it meets the requirements of the target task. The target task can be any image processing task, without specific limitations. Image processing tasks may include: human recognition, depth estimation, object classification, etc.

[0073] The target neural network can be the candidate subnetwork that has the highest accuracy on the target task among multiple candidate subnetworks.

[0074] In one embodiment, for each candidate subnetwork, the candidate subnetwork can be trained on a first training dataset corresponding to the target task, and its performance can be verified on a first verification dataset corresponding to the target task.

[0075] For example, taking the target task as a depth estimation task, the first training dataset may include multiple first sample images, each of which may correspond to a first reference depth map. The first reference depth map may include the first reference depth value corresponding to each pixel in the first sample image. The first reference depth value is the true value, which may be manually labeled or measured by a ranging device.

[0076] Training the candidate sub-network on the first training dataset includes: estimating the depth of a first sample image using the candidate sub-network to obtain a first estimated depth map output by the candidate sub-network. The first estimated depth map may include the first estimated depth value corresponding to each pixel in the first sample image. A training loss is calculated based on the first estimated depth value included in the first estimated depth map and the first reference depth value included in a first reference depth map. The parameters in the candidate sub-network are then adjusted inversely based on the training loss.

[0077] The process of depth estimation, loss calculation, and parameter tuning is continuously iterated. When the calculated training loss is lower than the set loss threshold, or the number of iterations exceeds the set number threshold, the training of the candidate sub-network on the first training dataset can be stopped, and the trained sub-network can be used to verify its performance on the first validation dataset.

[0078] The first verification dataset may include multiple first verification images, each of which may correspond to a first verification depth map. The first verification depth map may include the first verification depth value corresponding to each pixel in the first verification image. The first verification depth value is the true value, which may be manually marked or measured by a ranging device.

[0079] The performance of the trained candidate sub-network is validated on the first validation dataset, including: performing depth estimation on the first validation image using the trained candidate sub-network to obtain a second depth estimation map output by the trained candidate sub-network. The second depth estimation map may include the second estimated depth value corresponding to each pixel in the first validation image. Based on the second depth estimation value included in the second depth estimation map and the first validation depth value included in the first validation depth map, a first validation metric is calculated to determine the accuracy of the trained candidate sub-network's results on the depth estimation task.

[0080] Optionally, the first validation metric may include the L1 distance. The smaller the L1 distance, the higher the accuracy of the depth estimation result indicated by L1. For example, the L1 distance can be expressed using the following formula:

[0081] ;Formula (1);

[0082] Where L1_acc can be used to represent the L1 distance, T can be used to represent the total number of pixels, and i can represent the i-th pixel. It can be used to represent the second depth estimate. It can be used to represent the first verification depth value.

[0083] In this embodiment of the application, the performance of each trained candidate subnetwork can be verified to determine the optimal candidate subnetwork with the highest accuracy. The target neural network is then determined based on the optimal candidate subnetwork. The target neural network can be stored on electronic devices with limited computing power, such as smartphones and smart tablets, and used to perform any of the aforementioned target tasks for the electronic devices.

[0084] One approach is to directly identify the optimal candidate subnetwork as the target neural network; alternatively, the optimal candidate subnetwork can be pruned to compress it, and the compressed optimal candidate subnetwork can then be identified as the target neural network.

[0085] It should be noted that, in some embodiments, the electronic device can cyclically construct multiple candidate subnetworks and train and validate the performance of each candidate subnetwork. Each time a candidate subnetwork is constructed, it can be trained on the training dataset and its performance validated on the validation dataset.

[0086] In one embodiment, the electronic device may utilize a Recurrent Neural Network (RNN) as a controller, which executes the aforementioned step 210 to construct candidate subnetworks. Each time a candidate subnetwork is constructed, the controller can train and evaluate its performance, obtaining a validation metric. The accuracy indicated by the validation metric is then fed back to the controller to update the parameters in the RNN network. This allows the RNN to search for a better-performing target neural network from the first operator search space and the second search space, thus constructing the candidate subnetwork with the highest accuracy on the target task. The training and validation steps for the candidate subnetwork are then repeated until the accuracy of the candidate subnetwork exceeds a preset accuracy threshold.

[0087] As can be seen, the controller can construct multiple candidate subnetworks and determine the optimal candidate subnetwork from among them, thereby determining the target neural network based on the optimal candidate subnetwork.

[0088] In the aforementioned embodiments, different operator search spaces are designed based on the computing performance of different computing platforms, enabling the constructed target neural network to be applicable to both the first and second computing platforms, while achieving high accuracy on the target task. Simultaneously, the search for target network modules, the training of candidate subnetworks, and performance verification can all be automatically completed by electronic devices, thereby freeing researchers from the task of designing neural network structures and making neural network structure design more efficient; it may even uncover network structures that are difficult for humans to imagine, further improving network performance.

[0089] In some embodiments, the various hyperparameters related to the target neural network and its network structure can be searched and obtained based on the neural network search method disclosed in the foregoing embodiments. In other embodiments, in order to reduce the amount of search computation, a basic network framework can be predetermined. The basic network framework can define one or more hyperparameters related to the network structure of the neural network. Hyperparameters not defined in the basic network framework can be searched and obtained by the electronic device from the first operator search space and the second operator search space.

[0090] In other words, the target neural network can be obtained by supplementing the basic network framework by searching for target network modules from the first operator search space and the second operator search space. Since the target neural network is determined from multiple candidate subnetworks, each candidate subnetwork can also be constructed based on the basic network framework.

[0091] Therefore, each candidate subnetwork in step 210 above can be constructed through the following steps:

[0092] Based on the basic network framework, target network modules are searched from the first operator search space and the second operator search space, and the searched target network modules are used to supplement the basic network framework to obtain candidate subnetworks.

[0093] The basic network framework can define one or more hyperparameters related to the network structure, such as the number of network layers; or, the activation functions of the network layers can be defined, without any specific limitation.

[0094] In one embodiment, the hyperparameters defined by the basic network framework may include: the connectivity between adjacent network layers, and the dimensionality between the input and output data of each network layer.

[0095] It's important to note that the connection relationships defined in the basic network framework may not include the number of network layers; the size relationships defined in the basic network framework refer to the trend of size changes in the input and output data, rather than their absolute sizes. Data such as images or text can be converted into arrays before being input into the neural network. Therefore, the size of the input and output data can be represented by the size of the array.

[0096] For example, the basic network framework can define the network layers of the neural network as a chain structure, that is, adjacent network layers are interconnected without skip connections; or, the basic network framework can also define that the Mth network layer can be connected to the Yth network layer in addition to being connected to adjacent network layers. Here, M is a positive integer greater than or equal to 1, and Y is a positive integer greater than M.

[0097] The basic network framework can define that the size of the input data of the Mth network layer is greater than the size of the output data, where M can be a positive integer greater than or equal to 1. Alternatively, the basic network framework can further define that the size of the input data of the Mth network layer is twice the size of the output data.

[0098] Based on this, the basic network framework is supplemented by the searched target network modules. This can be achieved by searching for target network modules in the first operator search space and the second operator search space, and determining the type of each network layer in the basic network framework based on the searched target network modules. One target network module can correspond to one network layer. The number of target network modules searched from the aforementioned two operator search spaces can determine the number of network layers included in the neural network.

[0099] Alternatively, the target task processed by the constructed target neural network can be an image processing task, such as a depth estimation task or an object recognition task. Since U-nets exhibit excellent performance in image processing, the basic network framework can be determined as a U-net.

[0100] For example, please refer to Figure 5 , Figure 5 This is a structural example diagram of a U-shaped network framework disclosed in one embodiment. Figure 5 As shown, the network framework of the U-shaped network defines the interconnection between adjacent network layers, including a contraction path and an expansion path, and the network layers on the contraction path and the expansion path are symmetrical to each other in size.

[0101] From the first operator search space and the second operator search space, for example Figure 5 In the U-shaped network framework shown, each network layer randomly searches for a target network module to construct a candidate subnetwork.

[0102] Using the U-shaped network as the basic network framework for neural network search can, on the one hand, reduce the amount of computation required for the search and efficiently find neural networks that are applicable to different platforms; on the other hand, it can also leverage the advantages of the U-shaped network in the field of image processing, enabling the searched neural networks to achieve better performance in image processing tasks.

[0103] As can be seen, in the foregoing embodiments, the electronic device can construct candidate subnetworks using network modules in multiple operator search spaces. Different operator search spaces are designed based on the computing performance of different computing platforms, so that the final constructed target neural network can be applied to different computing platforms.

[0104] In one embodiment, the target neural network may need to run on a mobile terminal with limited computing power, such as a smartphone, or a small electronic device. If the target neural network is too large, it will be difficult for it to achieve good performance on the aforementioned mobile terminal. Based on this, the electronic device can prune the network to compress it.

[0105] Please see Figure 6 , Figure 6 This is a schematic diagram of the method flow of another neural network search method disclosed in one embodiment. For example... Figure 6 As shown, this method can be applied to any of the aforementioned electronic devices, without specific limitations. For example... Figure 6 As shown, the neural network search method may include the following steps:

[0106] 610. Construct multiple candidate subnetworks using the target network modules searched from the first operator search space and the second operator search space.

[0107] The first operator search space includes multiple different first network modules, which are designed based on the computing performance of the first computing platform; the second operator search space includes multiple different second network modules, which are designed based on the computing performance of the second computing platform.

[0108] The implementation method for the electronic device to search for target network modules from the first operator search space and the second operator search space to construct candidate subnetworks can refer to the foregoing embodiments, and will not be repeated below.

[0109] 620. Train multiple candidate sub-networks using the first training dataset, and verify the performance of each trained candidate sub-network using the first validation dataset to obtain the first validation metric for each trained candidate sub-network.

[0110] The first validation metric is used to indicate the accuracy of the trained candidate subnetwork on the target task. The target task can be any image processing task, without any specific limitation.

[0111] For example, the first verification metric can be the L1 distance. The smaller the value of the L1 distance, the higher the accuracy of the trained candidate subnetwork in the depth estimation task.

[0112] For example, the first training dataset may include a first sample image and a first reference depth map corresponding to the first sample image; the first verification dataset may include a first verification image and a first verification depth map corresponding to the first verification image.

[0113] In some embodiments, when the first verification dataset includes multiple first verification images and a first verification depth map corresponding to each first verification image, the electronic device can use each first verification image to perform performance verification on the trained candidate sub-network, thereby calculating the L1 distance corresponding to each first verification image, and calculating the average value of the L1 distances corresponding to multiple first verification images as the first verification metric of the trained candidate sub-network.

[0114] 630. Select the candidate subnetwork with the highest accuracy of the result indicated by the first verification metric from all trained candidate subnetworks as the optimal candidate subnetwork.

[0115] The optimal candidate subnetwork is the one with the highest accuracy among all trained candidate subnetworks, as indicated by the first validation metric. The target subnetwork can be obtained by pruning the optimal candidate subnetwork. Therefore, step 640 below can be performed.

[0116] 640. Prune the optimal candidate subnetwork and determine the compressed candidate subnetwork as the target neural network.

[0117] Pruning is a method for trimming trained networks. It typically involves finding an effective evaluation method to determine the importance of various parameters in the network, and then pruning parameters that are judged to be unimportant to reduce network redundancy and achieve network compression.

[0118] In the embodiments of this application, the pruning process for the optimal candidate subnetwork may include: channel pruning, weight pruning, neuron pruning, etc., and the specifics are not limited.

[0119] In one embodiment, channel pruning can be performed on the optimal candidate subnetwork, which refers to reducing the number of channels. The optimal candidate subnetwork includes convolutional layers, which are network layers where the target network module is a convolutional unit. For example, a CNN network may include multiple convolutional layers, and both the contraction and expansion paths of a U-shaped network can include convolutional layers. Each convolutional layer may correspond to a number of convolutional channels, which can be understood as the number of convolutional kernels in the convolutional layer. The number of convolutional channels corresponding to a convolutional layer can be greater than or equal to 1, without specific limitations.

[0120] Therefore, please refer to Figure 7 , Figure 7 This is a schematic flowchart of a method for pruning the optimal candidate subnetwork, as disclosed in one embodiment. Figure 7 The steps shown can be one implementation of step 640 described above, such as... Figure 7 The following steps may be included:

[0121] 710. Select one or more first convolutional layers from the best candidate subnetworks.

[0122] The first convolutional layer can be randomly selected from the best candidate subnetwork. The electronic device can select one convolutional layer from the best candidate subnetwork as the first convolutional layer each time, or it can select multiple convolutional layers as the first convolutional layer. There is no specific limitation.

[0123] 720. By reducing the number of convolutional channels corresponding to the first convolutional layer, one or more second convolutional layers are generated, thereby obtaining a channel subnetwork corresponding to each second convolutional layer.

[0124] For each selected first convolutional layer, the electronic device can reduce the number of convolutional channels corresponding to the first convolutional layer to generate one or more second convolutional layers.

[0125] For example, suppose the Xth convolutional layer is selected as the first convolutional layer from the optimal candidate subnetworks, and the Xth convolutional layer has 16 channels, where X is a positive integer greater than or equal to 1. The generated channel subnetwork can include the following two combinations:

[0126] Combination 1: If a second convolutional layer is generated based on the first convolutional layer, the number of convolutional channels in the Xth convolutional layer can be reduced from 16 to 8, resulting in a second convolutional layer with 8 channels, thus obtaining a channel subnetwork with 8 channels in the Xth convolutional layer.

[0127] Combination 2: If two second convolutional layers are generated based on the first convolutional layer, the number of convolutional channels in the Xth convolutional layer can be reduced from 16 to 8, resulting in a channel subnetwork with 8 channels in the Xth convolutional layer; and then, the number of convolutional channels in the Xth convolutional layer can be reduced from 16 to 4, resulting in a second convolutional layer with 4 channels, thus obtaining a channel subnetwork with 4 channels in the Xth convolutional layer.

[0128] In other words, if a first convolutional layer is selected from the optimal subnetwork, the number of channel subnetworks generated can be the same as the number of second convolutional layers.

[0129] For example, suppose two convolutional layers are selected from the optimal candidate subnetwork as the first convolutional layer, namely the Xth convolutional layer and the Yth convolutional layer in the optimal candidate subnetwork. The Xth convolutional layer has 16 channels, and the Yth convolutional layer has 32 channels. Y is a positive integer greater than or equal to 1 and different from X. Then the generated channel subnetwork can include the following three combinations:

[0130] Combination 1: If a second convolutional layer is generated based on each first convolutional layer, the number of convolutional channels of the Xth convolutional layer can be reduced from 16 to 8, and the number of convolutional channels of the Yth convolutional layer can be reduced from 32 to 16, resulting in two second convolutional layers and a channel subnetwork with 8 channels in the Xth convolutional layer and 16 channels in the Yth convolutional layer.

[0131] Combination 2: Generate a second convolutional layer based on the Xth convolutional layer, and generate multiple second convolutional layers based on the Yth convolutional layer (taking two as an example). Based on the Xth convolutional layer, a second convolutional layer with 8 channels can be generated, and based on the Yth convolutional layer, two second convolutional layers can be generated with 16 and 8 channels respectively. Therefore, two channel sub-networks can be obtained, as follows:

[0132] A channel subnetwork with 8 channels in the Xth convolutional layer and 16 channels in the Yth convolutional layer; and a channel subnetwork with 8 channels in the Xth convolutional layer and 8 channels in the Yth convolutional layer.

[0133] Combination 3: Generate multiple second convolutional layers based on each first convolutional layer. For example, generate two second convolutional layers for each of the Xth and Yth convolutional layers.

[0134] Two second convolutional layers are generated based on the Xth convolutional layer, with 8 and 4 channels respectively.

[0135] Two second convolutional layers are generated based on the Y-th convolutional layer, with 32 and 16 channels respectively. Therefore, a four-channel subnetwork can be obtained, as follows:

[0136] A channel subnetwork with 8 channels in the Xth convolutional layer and 32 channels in the Yth convolutional layer; a channel subnetwork with 8 channels in the Xth convolutional layer and 16 channels in the Yth convolutional layer; a channel subnetwork with 4 channels in the Xth convolutional layer and 32 channels in the Yth convolutional layer; a channel subnetwork with 4 channels in the Xth convolutional layer and 16 channels in the Yth convolutional layer.

[0137] When an electronic device reduces the number of convolutional channels corresponding to the first convolutional layer to generate the second convolutional layer, it can employ one or more different strategies, including: fixing the number of convolutional channels by a preset number or reducing it to one-Nth of the original number, where N is a positive integer; generating random numbers in real time and reducing the number of convolutional channels to that random number or to the specified value.

[0138] In one embodiment, one or more candidate channel numbers smaller than the first channel number can be generated based on the original number of first channels corresponding to the first convolutional layer in the optimal candidate subnetwork, thereby obtaining a channel number set.

[0139] The electronic device can further select one or more second channel numbers from the set of channel numbers. For each selected second channel number, the number of convolutional channels in the first convolutional layer can be replaced with the second channel number to obtain a second convolutional layer corresponding to each second channel number. For example, assuming the first channel number corresponding to the first convolutional layer is 16, the set of channel numbers may include 8 and 12. The electronic device can randomly select a candidate channel number from the set of channel numbers as the second channel number of the second convolutional layer.

[0140] In other words, the channel subnetwork can be obtained by replacing the number of convolutional channels corresponding to the first convolutional layer of the optimal candidate subnetwork with the number of second channels; the number of second channels is selected from the number of candidate channels included in the channel number set, and the number of each candidate channel is less than the number of first channels.

[0141] 730. Train each channel subnetwork using the second training dataset, and verify the performance of each trained channel subnetwork using the second validation dataset to obtain the second validation metric for each trained channel subnetwork.

[0142] The data in the second training dataset can be the same as the data in the first training dataset in terms of data type, and the data in the second validation dataset can also be the same as the data in the first validation dataset in terms of data type.

[0143] The second validation metric is used to indicate the accuracy of the trained channel subnetwork on the target task. For example, the second validation metric can also be the L1 distance; the smaller the value of the L1 distance, the higher the accuracy of the trained channel subnetwork on the target task.

[0144] 740. Among the trained channel subnetworks, the channel subnetwork with the highest accuracy indicated by the second verification metric is determined as the target neural network.

[0145] It should be noted that the electronic device can also iterate through the process of constructing multiple channel subnetworks and training and validating the performance of each channel subnetwork. Each time a channel subnetwork is constructed, it can be trained on a second training dataset and its performance validated on a second validation dataset.

[0146] The implementation method for training the channel sub-network on the second training dataset of the electronic device can refer to the implementation method for training the candidate sub-network on the first training dataset in the foregoing embodiments; the implementation method for validating the channel sub-network on the second validation dataset can refer to the implementation method for validating the candidate sub-network on the first validation dataset in the foregoing embodiments, and the following will not be repeated.

[0147] After performing performance verification on the channel subnetworks on the second verification dataset, a second verification index can be obtained for each channel subnetwork. Thus, the channel subnetwork with the highest accuracy indicated by the second verification index can be identified as the target neural network.

[0148] It can be seen that the target neural network can be the channel sub-network with the highest accuracy of the result indicated by the second verification metric among multiple channel sub-networks; each channel sub-network is obtained by reducing the number of convolutional channels corresponding to the first convolutional layer in the optimal candidate sub-network, each channel sub-network is trained using the second training dataset and its performance is verified using the second verification dataset to obtain the second verification metric of each channel sub-network.

[0149] In the aforementioned embodiments, the electronic device constructs candidate subnetworks using network modules searched from the first operator search space and the second operator search space. After training and verifying the optimal candidate subnetwork with the best performance, the optimal candidate subnetwork can be further pruned to compress the target neural network, making the target neural network more suitable for computing platforms with limited computing power, such as mobile phones.

[0150] The target neural network obtained based on the neural network search method disclosed in the foregoing embodiments can be used to perform the target task.

[0151] In one embodiment, the image processing task can be a depth estimation task, and the target neural network can be used to perform depth estimation on the image. The target neural network can be a depth estimation network.

[0152] Electronic devices can use a depth estimation network to estimate the depth of the image to be processed, and obtain a depth estimation image of the image to be processed as the image processing result. The depth estimation image includes the depth estimation value corresponding to each pixel in the image to be processed.

[0153] Optionally, after obtaining the depth estimation image of the image to be processed, the electronic device can determine the background region of the image to be processed, and determine the depth estimation value corresponding to the background region based on the depth estimation image. Furthermore, it can determine the blurring level of the background region based on the depth estimation value corresponding to the background region, and blur the background region according to the blurring level.

[0154] Please see Figure 8 , Figure 8 This is a schematic flowchart illustrating an image processing method disclosed in one embodiment. This image processing method can be applied to electronic devices with limited computing power, such as smartphones and tablets. Figure 8 As shown, the method may include the following steps:

[0155] 810. The trained depth estimation network is used to estimate the depth of the image to be processed, and the depth estimation image corresponding to the image to be processed is obtained.

[0156] The depth estimation network can be a target neural network obtained by searching based on any neural search network disclosed in the foregoing embodiments. It can be a candidate sub-network without model compression or a channel sub-network obtained after model compression. The specific method is not limited.

[0157] In the embodiments of this application, the depth-estimated image can be used for one or more different downstream image processing operations such as portrait segmentation and image blurring.

[0158] In one embodiment, the depth estimation image can be used for portrait segmentation. The electronic device can acquire the depth estimation image, which includes depth estimates corresponding to each pixel in the image to be processed. The electronic device can identify pixels in the image to be processed whose corresponding depth estimates are less than a depth threshold as belonging to the portrait region, and identify pixels in the image to be processed whose corresponding depth values ​​are greater than or equal to the depth threshold as belonging to the background region, thereby achieving accurate identification of the portrait region and the background region, and performing portrait segmentation on the image to be processed.

[0159] In one embodiment, the depth-estimated image can also be used for image blurring. The electronic device may perform steps 820-830.

[0160] 820. Determine the background region of the image to be processed, and determine the depth estimate corresponding to the background region based on the depth estimation image.

[0161] Electronic devices can perform portrait segmentation on the image to be processed based on depth estimation images to obtain the background region of the image to be processed. Alternatively, electronic devices can also identify the portrait region from the image to be processed based on portrait recognition algorithms that do not utilize depth values, such as portrait recognition or portrait matting, thereby determining the image region other than the portrait region as the background region.

[0162] After the background region is determined, the electronic device can obtain the depth estimate value corresponding to each pixel point included in the background region from the depth estimation image.

[0163] 830. Determine the blur level of the background region based on the depth estimate corresponding to the background region, and blur the background region according to the blur level.

[0164] The electronic device can determine the blur level of the background region based on the depth estimate corresponding to each pixel in the background region. Optionally, the blur level of the background region can be positively correlated with the depth estimate of the background region, that is, the larger the depth estimate of the background region, the higher the blur level and the more blurred the background.

[0165] Alternatively, the background region can be divided into two or more background sub-regions according to the depth estimate corresponding to each pixel. Each background sub-region can have its own blur level, and the blur level of the background sub-region can be positively correlated with the depth estimate of the background sub-region.

[0166] For example, if an electronic device blurs a background area using Gaussian blur, the blur level can be represented by the size of the Gaussian kernel. The Gaussian kernel can be represented by the following formula:

[0167] ;Formula (2);

[0168] in, (u,v) can be used to represent the Gaussian kernel, and (u,v) can be used to represent the image coordinates of a pixel. The standard deviation can be preset.

[0169] The higher the blur level, the larger the Gaussian kernel size, and the more blurred the background area after blurring; the lower the blur level, the smaller the Gaussian kernel size, and the clearer the background area after blurring.

[0170] It is evident that image depth estimation results play a crucial role in various application scenarios, such as portrait segmentation and image blurring. Therefore, lightweight depth estimation networks suitable for different computing platforms enable electronic devices with limited computing power to quickly and accurately estimate the depth of images to be processed. Taking image blurring as an example, lightweight depth estimation networks make real-time previewing of the blurring effect a reality. Furthermore, depth estimation networks can be directly stored and run on electronic devices, making offline image blurring possible, allowing users to perform offline operations on the images to be processed.

[0171] Please see Figure 9 , Figure 9 This is a schematic diagram of the structure of a neural network search device disclosed in one embodiment. This neural network search device can be applied to electronic devices such as cloud servers and personal computers, and is not specifically limited thereto. Figure 9 As shown, the neural network search device 900 may include a construction module 910 and a determination module 920.

[0172] The construction module 910 is used to construct multiple candidate subnetworks using target network modules searched from the first operator search space and the second operator search space; the first operator search space includes multiple different first network modules, which are designed based on the computing performance of the first computing platform; the second operator search space includes multiple different second network modules, which are designed based on the computing performance of the second computing platform; each network module includes a combination of one or more operators;

[0173] The determination module 920 is used to determine the target neural network based on multiple candidate subnetworks.

[0174] In one embodiment, the computing efficiency of the first network module on the first computing platform is higher than that on the second computing platform; the computing efficiency of the second network module on the second computing platform is higher than that on the first computing platform.

[0175] In one embodiment, the building module 910 may include a search unit and a building unit.

[0176] The search unit can be used to search for multiple target network modules from the first operator search space and the second operator search space based on the basic network framework; the basic network framework defines one or more hyperparameters of the neural network that are related to the network structure.

[0177] The building unit can be used to supplement the basic network framework with the searched target network modules to obtain the target subnetwork; the target subnetwork is any one of multiple candidate subnetworks.

[0178] In one embodiment, the hyperparameters defined by the basic network framework include: the connectivity between adjacent network layers, and the dimensionality between the input and output data of each network layer.

[0179] In one embodiment, the target task includes an image processing task, and the basic network framework is a U-shaped network framework.

[0180] In one embodiment, the determining module 920 may include a training unit, a verification unit, and a determining unit.

[0181] The training unit can be used to train multiple candidate sub-networks separately using the first training dataset;

[0182] The validation unit can be used to perform performance validation on each trained candidate sub-network using the first validation dataset to obtain the first validation index of each trained candidate sub-network; the first validation index is used to indicate the accuracy of the results of the trained candidate sub-network on the target task.

[0183] The determination unit can be used to determine the target neural network based on the optimal candidate subnetwork; the optimal candidate subnetwork is the candidate subnetwork with the highest accuracy of the result indicated by the first verification metric among all trained candidate subnetworks.

[0184] In one embodiment, the determining module 920 may include a compression unit, configured to prune the optimal candidate subnetwork. The aforementioned determining unit may also be configured to determine the compressed candidate subnetwork as the target neural network.

[0185] In one embodiment, the optimal candidate subnetwork includes a convolutional layer, and the target network module corresponding to the convolutional layer is a convolutional kernel.

[0186] The compression unit can also be used to select one or more first convolutional layers from the optimal candidate sub-networks; generate one or more second convolutional layers by reducing the number of convolutional channels corresponding to the first convolutional layers, thereby obtaining a channel sub-network corresponding to each second convolutional layer; train each channel sub-network using a second training dataset, and perform performance verification on each trained channel sub-network using a second verification dataset to obtain a second verification index for each trained channel sub-network. The second verification index is used to indicate the accuracy of the trained channel sub-network on the target task; and the channel sub-network with the highest accuracy indicated by the second verification index among the trained channel sub-networks is determined as the target neural network.

[0187] In one embodiment, the number of convolutional channels corresponding to the first convolutional layer is the number of first channels;

[0188] The compression unit can also be used to generate a set of channel numbers based on the first channel number, the set of channel numbers including one or more candidate channel numbers less than the first channel number; and to select one or more second channel numbers from each candidate channel number included in the set of channel numbers; and to replace the number of convolutional channels corresponding to the first convolutional layer with the second channel number to obtain a second convolutional layer corresponding to each second channel number.

[0189] In one embodiment, the neural network search device 900 may further include an estimation module. The aforementioned target neural network is used to perform depth estimation on an image, and the target neural network is a depth estimation network.

[0190] The estimation module can be used to perform depth estimation on the image to be processed through a depth estimation network to obtain a depth estimation image of the image to be processed; the depth estimation image includes the depth estimation value corresponding to each pixel in the image to be processed.

[0191] In one embodiment, the neural network search device 900 may further include a processing module.

[0192] The processing module can be used to determine the background region of the image to be processed, and determine the depth estimate value corresponding to the background region based on the depth estimate image; and determine the blur level of the background region based on the depth estimate value corresponding to the background region, and blur the background region according to the blur level.

[0193] As can be seen, in the foregoing embodiments, the neural network search device can design the operator search space based on the computational characteristics of different computing platforms, thereby enabling the searched target neural network to be applicable to different computing platforms. Furthermore, it can further perform automated pruning on the searched neural network, further compressing the complexity of the final generated target neural network, allowing the target neural network to run on electronic devices with limited computing power. Image processing based on the target neural network searched by the neural network search device can be compatible with the computing capabilities of different computing platforms, achieving high image processing accuracy on various platforms.

[0194] Please see Figure 10 , Figure 10 This is a schematic diagram of an image processing apparatus disclosed in one embodiment. This image processing apparatus can be applied to electronic devices such as smartphones and tablets. Figure 10 As shown, the image processing device 1000 may include a processing module 1010.

[0195] The processing module 1010 can be used to process the image to be processed using a target neural network to obtain the image processing result. The target neural network can be derived using any neural network search method disclosed in the foregoing embodiments, and is not specifically limited thereto.

[0196] In one embodiment, the target neural network may be a depth estimation network.

[0197] Processing module 1010 can be used to perform depth estimation on the image to be processed through a trained depth estimation network to obtain a depth estimation image corresponding to the image to be processed; the depth estimation image includes the depth estimation value corresponding to each pixel in the image to be processed.

[0198] In one embodiment, the depth-estimated image can be used for image blurring. Therefore, the image processing apparatus 1000 may further include a blurring module 1020.

[0199] The blurring module 1020 can be used to determine the background region of the image to be processed, and determine the depth estimate value corresponding to the background region based on the depth estimate image; and determine the blurring level of the background region based on the depth estimate value corresponding to the background region, and blur the background region according to the blurring level.

[0200] As can be seen, implementing this image processing device allows for accurate depth estimation of the image to be processed using a lightweight depth estimation network. Furthermore, the depth estimation network is applicable to different computing platforms; even if different image processing devices use different computing platforms for computation, accurate depth estimation results can still be obtained.

[0201] Please see Figure 11 , Figure 11 This is a schematic diagram of the structure of an electronic device disclosed in one embodiment. For example... Figure 11 As shown, the electronic device may include:

[0202] Memory 1110 storing executable program code;

[0203] Processor 1120 coupled to memory 1110;

[0204] Specifically, the processor 1120 calls the executable program code stored in the memory 1110 to execute any of the neural network search methods disclosed in the embodiments of this application; or, executes any of the image processing methods disclosed in the embodiments of this application.

[0205] It should be noted that, Figure 11 The electronic device shown may also include components not shown, such as a power supply, input buttons, camera, speaker, screen, RF circuit, Wi-Fi module, Bluetooth module, and sensors, which will not be described in detail in this embodiment.

[0206] This application discloses a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements any of the neural network search methods disclosed in this application.

[0207] This application discloses a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements any of the image processing methods disclosed in this application.

[0208] This application discloses a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and the computer program is operable to cause a computer to execute any of the neural network search methods disclosed in this application.

[0209] This application discloses a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and the computer program is operable to cause a computer to perform any of the image processing methods disclosed in this application.

[0210] It should be understood that the phrase "one embodiment" or "an embodiment" throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of this application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Those skilled in the art should also recognize that the embodiments described in the specification are optional embodiments, and the actions and modules involved are not necessarily essential to this application.

[0211] In the various embodiments of this application, it should be understood that the sequence number of each process does not necessarily imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0212] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; they can be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0213] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0214] If the integrated units described above are implemented as software functional units and sold or used as independent products, they can be stored in a computer-accessible memory. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several requests to cause a computer device (which can be a personal computer, server, or network device, specifically a processor in the computer device) to execute some or all of the steps of the methods described in the various embodiments of this application.

[0215] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, including read-only memory (ROM), random access memory (RAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), one-time programmable read-only memory (OTPROM), electrically-Erasable Programmable Read-Only Memory (EEPROM), compactdisc read-only memory (CD-ROM) or other optical disc storage, disk storage, magnetic tape storage, or any other computer-readable medium capable of carrying or storing data.

[0216] The foregoing has provided a detailed description of an image processing method, apparatus, electronic device, and storage medium disclosed in the embodiments of this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are merely for the purpose of helping to understand the method and its core ideas. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. An image processing method, characterized in that, include: The target neural network is used to process the image to be processed, and the image processing result is obtained. The target neural network is determined based on the optimal candidate subnetwork among multiple candidate subnetworks; each candidate subnetwork is constructed using target network modules searched from a first operator search space and a second operator search space; the optimal candidate subnetwork is the candidate subnetwork with the highest accuracy indicated by a first verification metric among the multiple candidate subnetworks; the first operator search space includes multiple different first network modules, which are designed based on the computing performance of a first computing platform; the second operator search space includes multiple different second network modules, which are designed based on the computing performance of a second computing platform; each network module includes a combination of one or more operators.

2. The method according to claim 1, characterized in that, The first network module has a higher computational efficiency on the first computing platform than on the second computing platform; the second network module has a higher computational efficiency on the second computing platform than on the first computing platform.

3. The method according to claim 1, characterized in that, The target neural network is obtained by supplementing the basic network framework with the target network module; the basic network framework defines one or more hyperparameters related to the network structure of the neural network.

4. The method according to claim 3, characterized in that, The hyperparameters defined in the basic network framework include: the connection relationships between adjacent network layers, and the size relationships between the input and output data of each network layer.

5. The method according to claim 1, characterized in that, Each candidate subnetwork is trained using a first training dataset, and the performance of the trained candidate subnetwork is verified using a first verification dataset to obtain the first verification metric for each candidate subnetwork.

6. The method according to claim 5, characterized in that, The target neural network is obtained by pruning the optimal candidate subnetwork.

7. The method according to claim 6, characterized in that, The optimal candidate sub-network includes: a convolutional layer, and the target network module corresponding to the convolutional layer is a convolutional kernel; The target neural network is the channel sub-network with the highest accuracy indicated by the second verification metric among multiple channel sub-networks; each channel sub-network is obtained by reducing the number of convolutional channels corresponding to the first convolutional layer in the optimal candidate sub-network; each channel sub-network is trained using a second training dataset and its performance is verified using a second verification dataset to obtain the second verification metric for each channel sub-network.

8. The method according to claim 7, characterized in that, In the optimal candidate subnetwork, the number of convolutional channels corresponding to the first convolutional layer is the number of first channels; The channel subnetwork is obtained by replacing the number of convolutional channels corresponding to the first convolutional layer of the optimal candidate subnetwork with the number of second channels; the number of second channels is selected from the candidate channel numbers included in the channel number set, and each candidate channel number is less than the number of first channels.

9. An image processing apparatus, characterized in that, include: The processing module is used to process the image to be processed using the target neural network to obtain the image processing result; The target neural network is determined based on the optimal candidate subnetwork among multiple candidate subnetworks; each candidate subnetwork is constructed using target network modules searched from a first operator search space and a second operator search space; the optimal candidate subnetwork is the candidate subnetwork with the highest accuracy indicated by a first verification metric among the multiple candidate subnetworks; the first operator search space includes multiple different first network modules, which are designed based on the computing performance of a first computing platform; the second operator search space includes multiple different second network modules, which are designed based on the computing performance of a second computing platform; each network module includes a combination of one or more operators.

10. An electronic device, characterized in that, The system includes a memory and a processor, wherein the memory stores a computer program that, when executed by the processor, causes the processor to perform the method as described in any one of claims 1 to 8.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 8.

Images