Image processing method, device, equipment and storage medium

Abstract

This application discloses an image processing method, apparatus, device, and storage medium, relating to the field of machine learning in artificial intelligence. The method includes: determining a reference network and a random network in a hypernetwork, wherein the hypernetwork includes at least two candidate operators, the reference network including a first set of operators predetermined within the hypernetwork, and the random network including a second set of operators randomly sampled within the hypernetwork, the hypernetwork being used for image processing; training and updating network parameters of the first and second sets of operators using a training image sample set; training and updating structural parameters of at least two candidate operators in the hypernetwork using a validation image sample set, the structural parameters being used to determine an image processing network from the hypernetwork; searching the hypernetwork for an image processing network based on the structural parameters; and invoking the image processing network to perform image processing on an input image to obtain an image processing result. This method can train the network structure of the image processing network and improve its network performance.

Description

Technical Field

[0001] The present application relates to the field of machine learning of artificial intelligence, and in particular to an image processing method, apparatus, device and storage medium. Background Art

[0002] With the booming development of deep learning, especially the development of neural networks, the method of using neural networks to process images has become more mature.

[0003] In related technologies, a training set of image samples is used to train a manually designed image processing network, enabling it to learn image processing capabilities and accurately output image processing results based on the input image. For example, for image classification tasks, the structure of the image classification network is manually designed: convolutional layers, pooling layers, fully connected layers, etc. Images from the training set are input into the image classification network to obtain predictions. The network parameters of the image classification network are then trained based on the loss between the predictions and the actual classification labels of the images.

[0004] The performance of image processing networks depends not only on network parameters but also on their structure. In related art, the network structure is manually designed based on experience. This limits network performance to the manually designed structure, preventing the optimal image processing network from being achieved. Summary of the Invention

[0005] The embodiments of the present application provide an image processing method, apparatus, device, and storage medium that can train the network structure of an image processing network and improve the network performance of the image processing network. The technical solution is as follows.

[0006] According to one aspect of the present application, there is provided an image processing method, the method comprising:

[0007] Determining a reference network and a random network in a hypernetwork, wherein the hypernetwork includes at least two candidate operators, the reference network includes a first group of operators predetermined in the hypernetwork, and the random network includes a second group of operators randomly sampled in the hypernetwork, wherein the hypernetwork is used for image processing;

[0008] Using a training image sample set to train and update network parameters of the first group of operators and the second group of operators;

[0009] Using a validation image sample set to train and update structural parameters of the at least two candidate operators in the super network, the structural parameters being used to determine an image processing network from the super network;

[0010] Searching the image processing network from the super network according to the structural parameters, wherein the image processing network includes a third group of operators in the super network;

[0011] The image processing network is called to perform image processing on the input image to obtain an image processing result.

[0012] In an optional embodiment, the method further includes:

[0013] Initializing the structural parameters of the supernetwork and the at least two candidate operators;

[0014] The network parameters of the at least two candidate operators in the super network are trained using a preheated image sample set.

[0015] In an optional embodiment, the using a preheated image sample set to train the network parameters of the at least two candidate operators in the super network includes:

[0016] determining a pre-warmed network in the super-network, the pre-warmed network comprising a fourth group of operators randomly sampled in the super-network;

[0017] Using the preheated image sample set to train and update the network parameters of the fourth group of operators;

[0018] Repeat the above steps to train the network parameters of the at least two candidate operators in the super network.

[0019] According to another aspect of the present application, there is provided an image processing apparatus, the apparatus comprising:

[0020] a determination module, configured to determine a reference network and a random network in a supernetwork, wherein the supernetwork includes at least two candidate operators, the reference network includes a first set of operators predetermined in the supernetwork, and the random network includes a second set of operators randomly sampled in the supernetwork, wherein the supernetwork is used for image processing;

[0021] A network training module, configured to train and update network parameters of the first group of operators and the second group of operators using a training image sample set;

[0022] a structure training module, configured to train and update structure parameters of the at least two candidate operators in the super network using a verification image sample set, wherein the structure parameters are used to determine an image processing network from the super network;

[0023] A search module, configured to search the image processing network from the super network according to the structural parameters, wherein the image processing network includes a third group of operators in the super network;

[0024] The calling module is used to call the image processing network to perform image processing on the input image to obtain an image processing result.

[0025] In an optional embodiment, the device further includes:

[0026] An initialization module, configured to initialize the structural parameters of the supernetwork and the at least two candidate operators;

[0027] A preheating module is used to train the network parameters of the at least two candidate operators in the super network using a preheating image sample set.

[0028] In an optional embodiment, the preheating module is configured to determine a preheating network in the supernetwork, wherein the preheating network includes a fourth group of operators randomly sampled in the supernetwork;

[0029] The preheating module is used to train and update the network parameters of the fourth group of operators using the preheating image sample set;

[0030] The preheating module is used to repeat the above steps to train the network parameters of the at least two candidate operators in the super network.

[0031] According to another aspect of the present application, a computer device is provided, comprising: a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set, or an instruction set, and the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the image processing method described above.

[0032] According to another aspect of the present application, a computer-readable storage medium is provided, wherein the storage medium stores at least one instruction, at least one program, a code set, or an instruction set, and the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement the image processing method described above.

[0033] According to another aspect of the embodiments of the present disclosure, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the image processing method provided in the aforementioned optional implementation.

[0034] The beneficial effects of the technical solutions provided in the embodiments of the present application include at least:

[0035] By setting up a supernetwork, setting up multiple candidate operators in the supernetwork, and having a computer device automatically train the supernetwork, the optimal operator combination constituting the image processing network is determined from the multiple candidate operators in the supernetwork. This allows the computer device to train the network structure of the image processing network, automatically finding the image processing network with the best performance from both the network structure and network parameter levels, and improving the network performance of the resulting image processing network. By pre-determining a reference network in the supernetwork, after randomly sampling a random network, the random network and the reference network are trained using a training image sample set. This allows not only the sampled candidate operators in the supernetwork to be trained, but also the candidate operators corresponding to the predetermined reference network to be trained. When training and updating structural parameters, the randomly sampled random network competes with the reference network, thereby preventing the adjustment of structural parameters from being biased solely towards the trained random network, and avoiding the problem of the strong getting stronger when training and determining the optimal network structure. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0037] Figure 1 is a block diagram of a computer device provided by an exemplary embodiment of the present application;

[0038] Figure 2 is a flowchart of an image processing method provided by another exemplary embodiment of the present application;

[0039] Figure 3 is a flowchart of an image processing method provided by another exemplary embodiment of the present application;

[0040] Figure 4 is a flowchart of an image processing method provided by another exemplary embodiment of the present application;

[0041] Figure 5 is a flowchart of an image processing method provided by another exemplary embodiment of the present application;

[0042] Figure 6 is a block diagram of an image processing apparatus provided by another exemplary embodiment of the present application;

[0043] Figure 7 is a structural diagram of a server provided by another exemplary embodiment of the present application;

[0044] Figure 8is a block diagram of a terminal provided by another exemplary embodiment of the present application. DETAILED DESCRIPTION

[0045] In order to make the objectives, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0046] Artificial Intelligence (AI) refers to the theories, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that seeks to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. AI also involves studying the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making.

[0047] Artificial intelligence (AI) technology is a comprehensive discipline encompassing a wide range of fields, encompassing both hardware and software technologies. Foundational AI technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0048] Machine learning (ML) is a multidisciplinary field that encompasses probability theory, statistics, approximation theory, convex analysis, and algorithmic complexity theory. It specifically studies how computers can simulate or implement human learning behaviors to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is at the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications span all areas of AI. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and learning through demonstration.

[0049] While neural network models built through machine learning can be adapted to diverse application scenarios and accomplish a wide range of tasks, their actual effectiveness, such as the time required for model training and the accuracy of model predictions, is closely related to the structure of the neural network model. Some neural network models may have complex structures but unsatisfactory prediction accuracy; conversely, some neural network models, despite having only a few layers, can achieve satisfactory prediction accuracy. The process of building a neural network model generally requires continuous trial and error by technicians with extensive experience in neural network structure design and parameter adjustment, which is labor-intensive and difficult to scale up.

[0050] With the emergence of a wide variety of neural network architectures, neural network architecture design is shifting from manual design to automated machine design. This has given rise to the field of Neural Architecture Search (NAS). Classic NAS methods use RNNs (Recurrent Neural Networks) as controllers to generate child networks, then train and evaluate these child networks to determine their performance (such as accuracy) and finally update the controller parameters. Neural Architecture Search has become a major research trend. While numerous methods have emerged, they generally encompass the following three key areas.

[0051] 1. Define the search space; that is, define the set of candidate network structures to be searched. Search spaces are broadly divided into global search spaces and cell-based search spaces. The former searches the entire network structure, while the latter searches only small structures, which are then stacked and spliced together to form a complete large network.

[0052] 2. Execute the search algorithm; the search algorithm refers to how to select in the search space. Depending on the method, the search algorithm can be roughly divided into three types: reinforcement learning-based method, evolutionary algorithm-based method and gradient-based method.

[0053] 3. Evaluate the performance of the sampled network; that is, evaluate the performance of the network structure on the target dataset.

[0054] A supernetwork is formed by aggregating all search spaces of a neural network architecture search model. Exemplarily, a supernetwork includes at least one network layer, each of which includes at least one candidate operator. A candidate operator is selected from each network layer in the supernetwork, and the candidate operators from the first to the last network layer are sequentially connected to form a subnetwork of the supernetwork. Each path in the supernetwork is a subnetwork.

[0055] The reference network is a subnetwork of the supernetwork. The reference network is a manually specified network, or / and the reference network is an image processing network obtained in a previous round of neural network architecture search. Exemplarily, the reference network is not a randomly sampled network, but a predetermined fixed network.

[0056] A random network is a subnetwork of a hypernetwork. A random network is a network randomly sampled from a hypernetwork. For example, the randomness can be random with equal probability or random with unequal probability based on the structural parameters of the candidate operators.

[0057] With the research and advancement of artificial intelligence technology, artificial intelligence technology has been studied and applied in many fields, such as common smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, unmanned driving, autonomous driving, drones, robots, smart medical care, smart customer service, etc. It is believed that with the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role.

[0058] The image processing solution provided in this application is highly versatile and practical. Specifically, the image processing method provided in this application is not restricted to specific scenarios. In scenarios such as face recognition and face detection, as long as a well-designed search method is employed, neural networks superior to those designed manually can be obtained. This method utilizes the concepts of multiple sampling and progressive optimization, ensuring a high probability of obtaining a new structure superior to the reference structure, and is highly practical.

[0059] Figure 1 A schematic diagram of a computer device 101 provided in an exemplary embodiment of the present application is shown. The computer device 101 may be a terminal or a server.

[0060] The terminal may include at least one of a smart phone, a laptop computer, a desktop computer, a tablet computer, a smart speaker, and a smart robot. In an optional implementation, a client supporting an image processing function is installed on the terminal.

[0061] Exemplarily, an image processing model is stored on the terminal. When the client needs to use the image processing function, the client can call the image processing model to complete the image processing.

[0062] Exemplarily, the image processing process may be completed by a terminal or a server.

[0063] The terminal and the server are connected to each other through a wired or wireless network.

[0064] The terminal includes a first memory and a first processor. An image processing model is stored in the first memory; the image processing model is called and executed by the first processor to implement the image processing method provided in the present application. The first memory may include but is not limited to the following: Random Access Memory (RAM), Read Only Memory (ROM), Programmable Read-Only Memory (PROM), Erasable Programmable Read-Only Memory (EPROM), and Electrically Erasable Programmable Read-Only Memory (EEPROM).

[0065] The first processor may be composed of one or more integrated circuit chips. Alternatively, the first processor may be a general-purpose processor, such as a central processing unit (CPU) or a network processor (NP). Alternatively, the first processor may implement the image processing method provided herein by running a program or code.

[0066] The server includes a second memory and a second processor. The second memory stores an image processing model; the image processing model is invoked by the second processor to implement the image processing method provided herein. Optionally, the second memory may include, but is not limited to, RAM, ROM, PROM, EPROM, and EEPROM. Optionally, the second processor may be a general-purpose processor, such as a CPU or NP.

[0067] Exemplarily, the computer device 101 stores an image processing model 102, and the image processing model 102 includes a super network 105. The super network 105 is formed by aggregating all possible network structures of the image processing model, and each path in the super network is a sub network. Exemplarily, the super network includes at least one network layer, and each network layer includes at least one candidate operator. When performing image processing, a candidate operator is selected from each network layer, and multiple candidate operators of multiple network layers form a sub network. For example, Figure 1 As shown, the super network may be composed of three network layers, each network layer includes three OPs (Operators), and each sub-network of the super network is composed of one candidate operator selected from each of the three network layers, that is, one sub-network includes three candidate operators.

[0068] Exemplarily, a candidate operator (or "operator") is a basic unit of neural network calculation. Exemplarily, a candidate operator includes one of the following: starting input, activation operator, two tensor calculations, multiple tensor calculations, normalization, feature extraction, and overfitting prevention; or, a candidate operator is composed of a combination of the following: starting input, activation operator, two tensor calculations, multiple tensor calculations, normalization, feature extraction, and overfitting prevention. Exemplarily, the above-mentioned starting input may include data. The activation operator may include at least one of the following: Relu function, LeakyRelu function, Relu6 function, Tan function, sigmoid function, and PRelu function. Two tensor calculations may include at least one of the following: + (addition), - (subtraction), * (multiplication), / (division), Add (addition), Sub (summation), Mul (multiplication), RealDiv function, BiasAdd (adding bias, bias is usually a one-dimensional tensor), MatMul (matrix multiplication), and Axpy (vector summation). Multiple Tensor calculations may include an EltwiseOp layer (used to perform one of the following operations: multiplication, addition, and maximum value on elements at corresponding positions of multiple Tensors). Normalization may include at least one of BatchNorm, LRN (Local Response Normalization), and SsdNormalize. Feature extraction includes at least one of Convolution, FullConnection, Correlation, DeConvolution, and DepConvolution. Anti-overfitting includes at least one of Pooling, Mean, ROIPooling, ROIAlign, and PSROIPooling.

[0069] Exemplarily, each candidate operator in the super network corresponds to a network parameter and a structure parameter.

[0070] Network parameters are the network parameters of a candidate operator used to perform neural network operations. Exemplarily, network parameters, also known as operation parameters, are the parameters of the neural network that are adjusted when training the neural network's performance using a sample set after the neural network's structure is determined. For example, when a candidate operator includes a convolution operation, the network parameters may be the value of the convolution kernel. Exemplarily, a computer device trains the network parameters of a candidate operator to ensure that the candidate operator has specified performance, thereby enabling the subnetwork to have specified recognition, classification, or prediction capabilities.

[0071] The structural parameters are used to select candidate operators from the hypernetwork to form the image processing network. Exemplarily, each candidate operator corresponds to a structural parameter. The computer device selects a candidate operator from each network layer in the hypernetwork based on the structural parameters. Multiple candidate operators from multiple network layers form the image processing network. Exemplarily, the structural parameters represent the probability of a candidate operator being selected; a larger structural parameter indicates a greater likelihood of selection.

[0072] like Figure 2 As shown, a flow chart of the image processing method provided by this application is given.

[0073] Step 201: The computer device receives an input training image sample set, a verification image sample set and specifies a reference network, for example, Figure 1 As shown, OP1 of the first network layer, OP1 of the second network layer, and OP1 of the third network layer in the super network can be designated as the reference network 103. Exemplarily, the sample set is divided into a training image sample set and a verification image sample set. For example, 80% of the sample data in the sample set is used as the training image sample set, and 20% of the sample data is used as the verification image sample set.

[0074] Step 202: The computer device initializes the structural parameters of the hypernetwork and candidate operators.

[0075] Step 203: The computer device preheats the supernetwork. To ensure that the candidate operators possess a certain level of image processing capability, the computer device preheats the network parameters of the candidate operators. For example, the computer device samples candidate operators from the supernetwork with equal probability to form subnetworks. The computer device then trains the subnetworks using a training image sample set, adjusting and updating their network parameters. The computer device repeats this step, continuously sampling and training the subnetworks until the candidate operators possess preliminary image processing capabilities.

[0076] Step 204: The computer device samples the random network and trains it. For example, the computer device randomly samples candidate operators in the super network according to the structural parameters to form a random network, and trains the random network using the training image sample set. For example, Figure 1 As shown, a computer device randomly samples OP2 of the first network layer, OP3 of the second network layer, and OP2 of the third network layer from the hypernetwork to form a random network 104. The random network is trained using a training image sample set to impart image processing capabilities. For example, the computer device extracts a batch of training sample data from the training image sample set and inputs it into the random network to obtain predicted values output by the random network. A loss function is used to calculate the loss between the predicted values and the true values, and gradients are obtained through backpropagation. The gradients are then used to update the network parameters of each candidate operator in the random network.

[0077] Step 205: The computer device trains the reference network. For example, the computer device trains the reference network using the training image sample set. Figure 1 As shown, a reference network 103 consisting of OP1 of the first network layer, OP1 of the second network layer, and OP1 of the third network layer is trained. Exemplarily, a batch of training sample data is taken from the training image sample set and input into the reference network to obtain the predicted value output by the reference network. The loss function is used to calculate the loss between the predicted value and the true value, and the gradient is obtained by backpropagation. The gradient is then used to update the network parameters of each candidate operator in the reference network.

[0078] Step 206: The computer device uses the validation image sample set to train the structural parameters of all candidate operators in the hypernetwork. For example, a batch of validation sample data is obtained from the validation set and input into the hypernetwork to obtain the predicted values output by the hypernetwork. A loss function is used to calculate the loss between the predicted values and the true values, and the structural parameters of the candidate operators are updated based on this loss.

[0079] Step 207: The computer device determines whether the hypernetwork has converged. If so, step 208 is executed; otherwise, step 204 is executed to perform iterative sampling and training. For example, the convergence condition can be arbitrary. For example, the convergence condition can be set as the number of iterations, the hypernetwork loss value being less than a threshold, or the image processing network being unchanged.

[0080] Step 208: After the super network converges, the computer device determines the image processing network based on the structural parameters of each candidate operator in the super network and outputs it.

[0081] For example, Figure 2 The method shown in is a method for performing a round of neural network structure search. Exemplarily, multiple rounds of neural network structure search can be performed, and the super network can be iteratively trained to obtain a network structure with better performance (the final image processing network).

[0082] like Figure 3 As shown, a flow chart of a method for executing the image processing method provided by this application is given.

[0083] Step 301: The computer device receives a designated reference network. This refers to the process of inputting a training image sample set and a validation image sample set, and designating a reference network, as in step 201. Exemplarily, the reference network designated in steps 301 and 201 is manually designated or automatically generated according to certain reference network generation rules. The reference network is designated before training the image processing model.

[0084] Step 302: The computer device searches for a neural network structure. Exemplarily, steps 202 to 208 are executed to perform a round of image processing to obtain a round of image processing network.

[0085] Step 303: The computer device obtains the image processing network of this round.

[0086] Step 304: The computer device determines whether the image processing network outputted in this round meets the business requirements. If so, the computer device proceeds to step 308; otherwise, the computer device proceeds to step 305.

[0087] Step 305: The computer device determines whether the image processing network outputted in this round is better than the reference network in this round. If so, the computer device proceeds to step 306; otherwise, the computer device proceeds to step 307.

[0088] Step 306: The computer device uses the image processing network output in this round as a reference network for the next round.

[0089] Step 307: The computer device uses the reference network of this round as the reference network of the next round, that is, keeps the reference network unchanged.

[0090] Step 308: The computer device outputs the image processing network that meets the business requirements as a final image processing network.

[0091] The method provided by the present application, when sampling sub-networks from a super-network for training, will additionally sample a manually designated sub-network as a reference network. The introduction of an additional reference network has the following benefits: 1) Alleviating the Matthew effect, since the training of the reference network is not limited to the structural parameters, even if the structural parameters of the candidate operator corresponding to the reference network are small, it can still be fully trained in each round of iteration, and then effectively participate in the competition of candidate operators in each network layer, and prevent the candidate operators with large structural parameters from becoming "stronger and stronger". 2) Alleviating the problem of model forgetting collapse, usually candidate operators with large structural parameters can often be sufficiently trained, but due to the drawbacks of the parameter sharing strategy, the network parameters of other sub-networks will be passively updated without minimizing the loss function, which will lead to model forgetting collapse. After sampling an additional reference network structure, not only the current "optimal" candidate operator can be trained, but the candidate operators of the reference network can also be sufficiently trained, and their performance can be guaranteed. 3) Improve the stability of search results. When different candidate operators compete, the additional reference network has been fully trained and is quite competitive. The search process will inevitably proceed in the direction of being better than the reference network. Therefore, the searched image processing network is likely to be better than the reference substructure, and the stability of the search results is guaranteed.

[0092] Furthermore, the image processing method provided by this application does not achieve the desired result overnight by obtaining an image processing network. Instead, the method continuously uses the image processing network obtained in the previous round of search as the reference network for the next round of search, and gradually iterates until the desired final image processing network is obtained. The reference network for the first round of iteration can be a manually designed network. This method does not shortsightedly take one round of search results as the final result, but rather continuously optimizes and fully utilizes the better intermediate network structures, performs iterative searches, and improves the performance of the final output network structure.

[0093] Figure 4 FIG1 shows a flow chart of an image processing method provided by an exemplary embodiment of the present application. The method can be executed by a computer device, for example, Figure 1 The method is performed by the computer device shown in FIG.

[0094] Step 401: Determine a reference network and a random network in a hypernetwork, wherein the hypernetwork includes at least two candidate operators, the reference network includes a first set of operators predetermined in the hypernetwork, and the random network includes a second set of operators randomly sampled in the hypernetwork, and the hypernetwork is used for image processing.

[0095] Exemplarily, the reference network is a predetermined network, for example, a manually specified network. The random network is a network obtained by random sampling. Exemplarily, the sampling of the random network can be completely random sampling (equal probability sampling) or random sampling based on the structural parameters of each candidate operator. Exemplarily, both the reference network and the random network are subnetworks of a supernetwork, and are composed of at least one candidate operator in the supernetwork.

[0096] The candidate operators in the hypernetwork are used to perform at least one image processing related operation on the input image or data.

[0097] The first group of operators includes at least one candidate operator in the hypernetwork. The second group of operators includes at least one candidate operator in the hypernetwork.

[0098] Exemplarily, the reference network may be a subnetwork of a hypernetwork, or may refer to multiple subnetworks of a hypernetwork. Exemplarily, the random network may be a subnetwork of a hypernetwork, or may refer to multiple subnetworks of a hypernetwork.

[0099] A hypernetwork is a network used to process an input image and generate an image processing result. Image processing includes at least one of image classification, object detection, and image segmentation. For example, a hypernetwork is a neural network used for face recognition or face detection.

[0100] Step 402: Use the training image sample set to train and update the network parameters of the first group of operators and the second group of operators.

[0101] Exemplarily, the random network and the reference network are trained using the training image sample set, so that the random network and the reference network have image processing capabilities. Exemplarily, the training sample data in the training image sample set includes input data and true values, and the network parameters of the random network and the reference network are trained using the training image sample set, so that the random network and the reference network can output corresponding true values based on the input data.

[0102] The data in the training image sample set can be obtained according to specific business needs. For example, the training image sample set includes sample images and true values corresponding to the sample images. The true values corresponding to the sample images are the true image processing results obtained after image processing on the sample images.

[0103] For example, when using the image processing method to search for the network structure of an image classification network, the training sample data in the training image sample set may include images (input data) and classification labels (true values) corresponding to the images, so that the random network and the reference network can be trained using the training image sample set so that they can output the classification label of the image based on the input image.

[0104] For another example, when the image processing method is used to search for the network structure of an image segmentation network, the training sample data in the training image sample set may include an image (input data) and an image segmentation result (true value) corresponding to the image, so that the training image sample set can be used to train a random network and a reference network so that they can output the image segmentation result of the image based on the input image.

[0105] For another example, when using the image processing method to search for the network structure of the target detection network, the training sample data in the training image sample set may include an image (input data) and a target position (true value) of the target in the image, so that the random network and the reference network can be trained using the training image sample set so that they can output the target position of the target in the image based on the input image.

[0106] Step 403: Use the verification image sample set to train and update the structural parameters of at least two candidate operators in the super network. The structural parameters are used to determine the image processing network from the super network.

[0107] Exemplarily, after training the reference network and the random network, the structural parameters of the hypernetwork are trained using a validation image sample set. Exemplarily, the validation sample data in the validation image sample set includes input data and true values, and the structural parameters of the hypernetwork are trained using the validation image sample set so that the hypernetwork can output corresponding true values based on the input data. Exemplarily, the network parameters of each candidate operator in the hypernetwork are fixed unchanged, and after input data is input into the hypernetwork, the candidate operators output predicted values based on the input and output, and the total output of the hypernetwork is the weighted calculation result of the predicted values output by each candidate operator according to the structural parameters. By adjusting the structural parameters of each candidate operator, the total output of the hypernetwork can be adjusted, thereby making the total output close to the true value.

[0108] Exemplarily, the training image sample set and the verification image sample set are both sample data sets, and the data types of the two are the same, but the specific data therein can be the same or different. For example, the two can be the same sample data set or different sample data sets.

[0109] Step 404: Search and obtain an image processing network from the hypernetwork according to the structural parameters. The image processing network includes the third group of operators in the hypernetwork.

[0110] Exemplarily, the super network includes at least one network layer, each network layer includes at least one candidate operator. Exemplarily, the computer device selects the candidate operator with the largest structural parameter in each network layer to form an image processing network.

[0111] Step 405: Call the image processing network to perform image processing on the input image to obtain an image processing result.

[0112] Exemplarily, after the image processing method provided in this embodiment is used to search for the optimal network structure: the image processing network, the image processing network can be called to perform image processing on the image.

[0113] For example, an image processing network is called to perform image classification on an input image to obtain an image classification result, which includes a classification label of the input image.

[0114] For another example, the image processing network is called to perform target detection on the input image to obtain a target detection result, which includes the target position (target box) of the detected target on the input image.

[0115] For another example, the image processing network is called to perform image segmentation on the input image to obtain an image segmentation result, which includes a classification label for each pixel in the input image.

[0116] For example, if the business requirement is image classification, the image processing network obtained has image classification capabilities. After obtaining the image processing network, the image processing network can be called to classify the input image to obtain the image classification result.

[0117] In summary, the method provided in this embodiment sets up a supernetwork, sets up multiple candidate operators in the supernetwork, and automatically trains the supernetwork by a computer device, thereby determining the optimal operator combination for constituting an image processing network from the multiple candidate operators in the supernetwork. This allows the computer device to train the network structure of the image processing network, automatically searching for the image processing network with the best performance from both the network structure and network parameter levels, and improving the network performance of the resulting image processing network. By pre-determining a reference network in the supernetwork, after randomly sampling a random network, the random network and the reference network are trained using a training image sample set. This allows not only the sampled candidate operators in the supernetwork to be trained, but also the candidate operators corresponding to the predetermined reference network to be trained. When training and updating structural parameters, the randomly sampled random network competes with the reference network, thereby preventing the adjustment of structural parameters from being biased solely towards the trained random network, and avoiding the problem of the strong getting stronger when training and determining the optimal network structure.

[0118] For example, an exemplary embodiment of obtaining a final image processing network through multiple rounds of searching is given. The final image processing network is a sub-network that meets the business requirements (meets the requirement conditions).

[0119] Figure 5 FIG1 shows a flow chart of an image processing method provided by an exemplary embodiment of the present application. The method can be executed by a computer device, for example, Figure 1 The method is executed by the terminal or server shown in the figure. The method includes the following steps.

[0120] Step 4011: Receive first indication information, where the first indication information is used to indicate a first group of operators constituting a reference network.

[0121] For example, the reference network may be a manually specified network. The computer device may receive the input instruction information and determine the reference network according to the candidate operators indicated in the instruction information.

[0122] Exemplarily, before step 4011, or after step 4012 and before step 4013, the hypernetwork is initialized. That is, the structural parameters of the hypernetwork and at least two candidate operators are initialized; and the network parameters of at least two candidate operators in the hypernetwork are trained using the preheated image sample set.

[0123] Exemplarily, a method for preheating a supernetwork using a preheated image sample set may be: determining a preheated network in the supernetwork, the preheated network including a fourth group of operators randomly sampled in the supernetwork; using the preheated image sample set to train and update network parameters of the fourth group of operators; and repeating the above steps to train network parameters of at least two candidate operators in the supernetwork.

[0124] For example, the warm-up of the hypernetwork only changes the network parameters of the candidate operators, without changing the structural parameters. Through prediction, the candidate operators can have preliminary performance.

[0125] Exemplarily, the warm-up image sample set may be the same sample data set as the training image sample set, or may be a different sample data set.

[0126] Step 4012: Determine a reference network in the supernetwork according to the first indication information.

[0127] Exemplarily, the supernetwork includes at least one network layer, each network layer includes at least one candidate operator. The computer device selects a candidate operator from each network layer of the supernetwork according to the first indication information to obtain a first group of operators to form a reference network.

[0128] Step 4013: Determine a random network in the super network.

[0129] Exemplarily, the computer device randomly samples with equal probability in the super network to obtain the second group of operators to form a random network; or, based on the structural parameters of at least two candidate operators, randomly sample in the super network to obtain the second group of operators to form a random network.

[0130] For example, the sampling of the random network can be completely random, that is, all candidate operators in each network layer have equal probability of being selected.

[0131] For example, random network sampling can also be performed based on structural parameters. That is, the probability of each candidate operator being selected will be different depending on the structural parameters of the candidate operator.

[0132] Exemplarily, the supernetwork includes at least one network layer, each network layer including at least one candidate operator. The computer device invokes a sampling function to calculate a one-hot vector for each network layer based on structural parameters of the at least one candidate operator in each network layer. Based on the one-hot vector for each network layer, the computer device selects a candidate operator from each network layer to obtain a second group of operators to form a random network.

[0133] For example, the sampling function can output a one-hot vector based on multiple structural parameters of multiple candidate operators in each network layer, and the candidate operator corresponding to 1 in the one-hot vector is the selected candidate operator. For example, the sampling function can be a GumbelSoftmax function.

[0134] Step 402: Use the training image sample set to train and update the network parameters of the first group of operators and the second group of operators.

[0135] Exemplarily, the computer device inputs the first sample in the training image sample set into the random network to obtain a first prediction value; updates the network parameters of the second group of operators based on the loss training between the first prediction value and the first true value, where the first true value is the true value corresponding to the first sample; inputs the second sample in the training image sample set into the reference network to obtain a second prediction value; updates the network parameters of the first group of operators based on the loss training between the second prediction value and the second true value, where the second true value is the true value corresponding to the second sample.

[0136] Step 403: Use the verification image sample set to train and update the structural parameters of at least two candidate operators in the super network. The structural parameters are used to determine the image processing network from the super network.

[0137] Exemplarily, the computer device inputs the third sample in the verification image sample set into the hypernetwork to obtain a third prediction value; and updates the structural parameters of at least two candidate operators in the hypernetwork based on the loss training of the third prediction value and the third true value, where the third true value is the true value corresponding to the third sample.

[0138] Exemplarily, the super network includes m network layers, the first network layer includes q candidate operators, and the jth network layer includes k candidate operators, where m, q, and k are positive integers and j is an integer greater than 1. The method for the super network to output the third prediction value may be: inputting the third sample in the verification image sample set into the q candidate operators in the first network layer respectively to obtain q intermediate outputs; using the structural parameters corresponding to the q candidate operators as weights, calculating the weighted sum of the q intermediate outputs, and obtaining the first output of the first network layer. Inputting the j-1th output of the j-1th network layer into the k candidate operators in the jth network layer respectively to obtain k intermediate outputs; using the structural parameters corresponding to the k candidate operators as weights, calculating the weighted sum of the k intermediate outputs, and obtaining the jth output of the jth network layer; repeating this step to determine the mth output of the mth network layer as the third prediction value.

[0139] For example, the output of a network layer in the hypernetwork is:

[0140]

[0141] Among them, x l+1 is the output of the l+1th network layer, x l is the output of the lth network layer, x0 is the third sample, b l+1,i is the i-th candidate operator in the l+1-th network layer, b l+1,i (x l) is x l Input b l+1,i The output obtained after l+1,i is the coefficient of the i-th candidate operator in the l+1-th network layer, m l+1,i It takes 0 or 1 and is calculated as follows:

[0142]

[0143] Among them, α l is the structural parameter of the lth network layer, α l,i is α l The structural parameters of the i-th candidate operator in g; l,i is the Gumbel noise, which is an independent and identically distributed standard Gumbel distributed random variable; τ is the temperature coefficient, which is a constant.

[0144] Right now,

[0145] Among them, α is the structural parameter, w is the network parameter, L train is the loss of training network parameters, L val is the loss of training structural parameters. The above formula can be understood as: keeping the structural parameters unchanged, training the network parameters of the random network / reference network to L train Then, based on the trained network parameter book, the validation set is used to train the structural parameters of the hypernetwork to L val The minimum value is taken to obtain the trained structural parameters. Exemplarily, these two steps are repeated alternately until the super network converges.

[0146] Exemplarily, steps 4013 to 403 are repeatedly performed to iteratively train the hypernetwork until convergence, and then step 404 is performed.

[0147] Step 404: Search and obtain an image processing network from the hypernetwork according to the structural parameters. The image processing network includes the third group of operators in the hypernetwork.

[0148] Exemplarily, steps 401 to 404 can be executed separately to output the image processing network, that is, only one round of neural network structure search is performed to obtain the image processing network. Exemplarily, after one round of neural network structure search image processing is completed, steps 501 to 506 can also be executed to determine whether the obtained image processing network meets the business requirements. If not, the next round of search can be performed, and the neural network structure search is iteratively performed until the final image processing network that meets the business requirements is found. That is, steps 401 to 404 are the first round of neural network structure search, and steps 501 to 505 are the i-th round of neural network structure search. Assuming that the image processing network finally obtained in the n-th round of neural network search meets the business requirements, it is output as the final image processing network in step 506.

[0149] Step 501: Determine whether the image processing network of round i-1 meets the requirements. If so, proceed to step 506; otherwise, proceed to step 502.

[0150] Exemplarily, the image processing network mentioned in steps 401 to 404 is the first-round image processing network, the reference network mentioned is the first-round reference network, and the random network mentioned is the first-round random network.

[0151] Requirements can be set arbitrarily. They are used to determine the image processing capabilities of the image processing network. Depending on the business / task of the neural network, you can set corresponding requirements to determine whether the searched neural network structure meets the business requirements. For example, for a classification neural network model, a requirement might be that the model's recognition accuracy exceeds a threshold.

[0152] Step 502: In response to the i-1th round image processing network not meeting the requirement conditions, an i-th round reference network and an i-th round random network are determined in the super network, where the i-th round reference network includes a first group of i-th round operators predetermined in the super network, and the i-th round random network includes a second group of i-th round operators randomly sampled in the super network, where i is an integer greater than 1 and less than n.

[0153] For example, if the image processing network selected in the first round does not meet the requirements, the next round of search can be continued.

[0154] In the next round, the reference network can directly use the reference network of the previous round, or select a better one from the reference network of the previous round and the image processing network, or manually specify another reference network.

[0155] Exemplarily, a method for determining the i-th round reference network may be: calculating performance parameters of the i-1th round image processing network and the i-1th round reference network, where i is an integer greater than 1 and not greater than n; and determining the network with better performance parameters between the i-1th round image processing network and the i-1th round reference network as the i-th round reference network.

[0156] For example, the performance parameter is used to describe the quality of the network. Any performance measurement method can be used to measure the quality of the reference network and the image processing network, and the better network is determined as the reference network for the next round.

[0157] Exemplarily, the method for determining the i-th round reference network may also be: receiving second indication information, where the second indication information is used to indicate the i-th round first group of operators constituting the i-th round reference network; and determining the i-th round reference network in the supernetwork according to the second indication information.

[0158] Exemplarily, the second indication information can be the same as the first indication information, i.e., the reference network for each search round is indicated once when the image processing model is initialized. Exemplarily, the second indication information can also be different from the first indication information. That is, the indication information is received before each search round to determine the reference network for that round.

[0159] Step 503: Use the training image sample set to train and update the network parameters of the first group of operators in the i-th round and the second group of operators in the i-th round.

[0160] For example, the training of network parameters may refer to step 402 .

[0161] Step 504: Use the verification image sample set to train and update the structural parameters of at least two candidate operators in the hypernetwork.

[0162] For example, the training of the structural parameters may refer to step 403 .

[0163] Step 505: Search and obtain the i-th round image processing network from the super network according to the structural parameters. The i-th round image processing network includes the i-th round third group of operators in the super network.

[0164] Exemplarily, the determination of the image processing network may refer to step 404 .

[0165] Step 506, repeat the above steps until the n-th round image processing network meets the required conditions, and output the n-th round image processing network as the final image processing network, where n is a positive integer.

[0166] For example, after each round of search, it is determined whether the image processing network output in this round meets the required conditions. If so, it is used as the final image processing network output. If not, the next round of search is continued.

[0167] Step 507: Call the final image processing network to perform image processing on the input image to obtain an image processing result.

[0168] In summary, the method provided in this embodiment, when sampling subnetworks from a supernetwork for training, additionally samples a designated subnetwork as a reference network. By training the reference network, the Matthew effect is mitigated. Because the training of the reference network is not limited by structural parameters, even if the structural parameters of the candidate operator corresponding to the reference network are relatively small, it can still be fully trained in each iteration, thereby effectively participating in the competition among candidate operators at each network layer, preventing the "stronger-than-stronger" candidate operators with large structural parameters. By additionally sampling and training a reference network structure, not only can the current "optimal" candidate operator be trained, but the candidate operators of the reference network can also be adequately trained, thereby ensuring the performance of the candidate operators. By training the reference network, when different candidate operators compete, since the additional designated reference network has been fully trained and has considerable competitiveness, the search process will inevitably proceed in the direction of being superior to the reference network. Therefore, the image processing network searched is likely to be superior to the reference substructure, and the stability of the search results is guaranteed.

[0169] The method provided in this embodiment does not achieve the desired image processing network in one go. Instead, the method iterates by continuously using the image processing network from the previous search as a reference network for the next search, until a desired final image processing network is obtained. The reference network for the first iteration can be a manually designed network. This method does not short-sightedly consider a single round of search results as the final result. Instead, it continuously optimizes and fully utilizes the optimal intermediate network structures, conducting iterative searches to improve the performance of the final output network structure.

[0170] The image processing method provided in this application does not limit the usage scenarios. In scenarios such as face recognition and face detection, as long as a reasonable search method is designed, a neural network structure that is better than the one designed manually can be searched.

[0171] For example, an exemplary embodiment of applying the image processing method provided by the present application in a face recognition scenario is given.

[0172] First, a training image sample set and a verification image sample set are obtained. The sample data in the training image sample set and the verification image sample set include sample images (face images and non-face images) and labels corresponding to the sample images (face or non-face).

[0173] Designing a hypernetwork of an image processing model. For example, manually designing candidate operators for each network layer in the hypernetwork to form the hypernetwork.

[0174] Manually specify the reference network in the supernetwork.

[0175] The computer device receives input training image sample sets, verification image sample sets, a hypernetwork, and a specified reference network, and initializes network parameters and structural parameters of the hypernetwork.

[0176] The computer device uses the training image sample set to preheat each candidate operator in the supernetwork. For example, a subnetwork is randomly sampled from the supernetwork with equal probability. The subnetwork is trained using the training image sample set to output whether an input sample image is a face. This process is repeated to iteratively train the supernetwork, enabling the candidate operators in the supernetwork to have preliminary face recognition capabilities.

[0177] The computer device randomly samples the structural parameters from the hypernetwork to obtain a random network, and uses the training image sample set to train the network parameters of the random network so that it can recognize whether the input sample image is a human face.

[0178] The computer device uses the training image sample set to train the network parameters of the reference network so that it can identify whether the input sample image is a human face.

[0179] The computer device uses the verification image sample set to train the structural parameters of the hypernetwork, so that the hypernetwork can identify whether the input sample image is a human face.

[0180] Determine whether the supernetwork has converged. If not, repeat the steps of sampling a random network, training the random network, and training the reference network to iteratively train the network parameters and structural parameters of the supernetwork. If converged, output an image processing network based on the structural parameters of the current supernetwork. Exemplarily, the convergence condition can be determining whether the image processing network output from this training is identical to the image processing network output from the previous training. If so, the supernetwork is determined to have converged; if not, the supernetwork is determined to have not converged.

[0181] Exemplarily, executing the above steps can complete one round of neural network structure search, obtaining one round of image processing network, which can output whether the input image is a face image. Exemplarily, multiple rounds of neural network structure search can also be performed.

[0182] The computer device determines whether the image processing network output in the previous round meets the business requirements. If so, the image processing network output in the previous round is determined as the final image processing network. If not, the next round of neural network structure search is continued.

[0183] For example, the computer device compares the performance of the image processing network from the previous round with the performance of the reference network from the previous round, and determines the network with the better performance as the reference network for the next round. For example, the computer device may determine the accuracy of two networks in recognizing faces and determine the one with the higher accuracy as the reference network for the next round.

[0184] The above steps of sampling random networks, training random networks, training reference networks, and determining convergence are then repeated. The image processing network for this round is output, and it is determined whether the image processing network for this round meets the business requirements. The super network is iteratively trained in this way until the output image processing network meets the business requirements, resulting in the final image processing network. For example, the business requirement condition can be set arbitrarily, for example, the business requirement condition can be set to an accuracy rate greater than a threshold.

[0185] For example, an exemplary embodiment of applying the image processing method provided by the present application to an image classification scenario is given.

[0186] In the first step, a computer device determines a reference network and a random network in a hypernetwork, where the hypernetwork includes at least two candidate operators, the reference network includes a first set of operators predetermined in the hypernetwork, and the random network includes a second set of operators randomly sampled in the hypernetwork, and the hypernetwork is used for image classification.

[0187] The computer device receives first indication information, where the first indication information is used to indicate a first group of operators constituting a reference network; the computer device determines a reference network in the supernetwork based on the first indication information. Exemplarily, the supernetwork includes at least one network layer, each network layer includes at least one candidate operator, and the computer device selects a candidate operator from each network layer of the supernetwork based on the first indication information to obtain a first group of operators to constitute the reference network. Exemplarily, the input data of each layer in the supernetwork is the output data of the previous layer, the input data of the first network layer is the input image, and the output data of the last network layer is the image processing result, i.e., the image classification result.

[0188] Exemplarily, the reference network can be a network determined based on the developer's experience. For example, a first group of operators is determined from a super network based on the network structure of an existing image classification network with better performance. The network structure composed of the first group of operators is the same as the network structure of the existing image classification network.

[0189] Computer devices randomly sample with equal probability in the super network to obtain a second group of operators to form a random network; or, based on the structural parameters of at least two candidate operators, randomly sample in the super network to obtain a second group of operators to form a random network.

[0190] Exemplarily, the computer device calls a sampling function to calculate the one-hot vector of each network layer based on the structural parameters of at least one candidate operator of each network layer; according to the one-hot vector of each network layer, a candidate operator is selected from each network layer to obtain a second group of operators to form a random network.

[0191] Exemplarily, before executing the first step, the computer device will also initialize the structural parameters of the super network and at least two candidate operators; use the preheated image sample set to train the network parameters of at least two candidate operators in the super network, so that each operator in the super network has preliminary image classification capabilities.

[0192] In the second step, the computer device uses the training image sample set to train and update the network parameters of the first group of operators and the second group of operators.

[0193] Exemplarily, the training image sample set includes sample images and true values corresponding to the sample images, where the true values are classification labels of the sample images.

[0194] The computer device uses a training image sample set to train the first group of operators and the second group of operators respectively, so that the reference network composed of the first group of operators can output relatively accurate image classification results based on the input image; and the random network composed of the second group of operators can output relatively accurate image classification results based on the input image.

[0195] For example, the computer device inputs the first sample in the training image sample set into the random network to obtain a first predicted classification label; updates the network parameters of the second group of operators based on the loss training between the first predicted classification label and the first true classification label, where the first true classification label is the true classification label corresponding to the first sample; inputs the second sample in the training image sample set into the reference network to obtain a second predicted classification label; updates the network parameters of the first group of operators based on the loss training between the second predicted classification label and the second true classification label, where the second true classification label is the true classification label corresponding to the second sample.

[0196] In the third step, the computer device uses the verification image sample set to train and update the structural parameters of at least two candidate operators in the super network, and the structural parameters are used to determine the image classification network from the super network.

[0197] The computer device inputs the third sample in the verification image sample set into the hypernetwork to obtain a third predicted classification label; and updates the structural parameters of at least two candidate operators in the hypernetwork based on the loss training of the third predicted classification label and the third true classification label, where the third true classification label is the true classification label corresponding to the third sample.

[0198] Exemplarily, the super network includes m network layers, the first network layer includes q candidate operators, and the jth network layer includes k candidate operators, where m, q, and k are all positive integers, and j is an integer greater than 1. The computer device inputs the third sample in the verification image sample set into the q candidate operators in the first network layer, respectively, to obtain q intermediate outputs; uses the structural parameters corresponding to the q candidate operators as weights, calculates the weighted sum of the q intermediate outputs, and obtains the first output of the first network layer; inputs the j-1th output of the j-1th network layer into the k candidate operators in the jth network layer, respectively, to obtain k intermediate outputs; uses the structural parameters corresponding to the k candidate operators as weights, calculates the weighted sum of the k intermediate outputs, and obtains the jth output of the jth network layer; repeats this step, and determines the mth output of the mth network layer as the third predicted classification label.

[0199] In the fourth step, the computer device searches the hypernetwork according to the structural parameters to obtain an image classification network, where the image classification network includes the third group of operators in the hypernetwork.

[0200] In the fifth step, the computer device calls the image classification network to classify the input image and obtain the image classification result.

[0201] Exemplarily, executing steps 1 to 4 is to perform a round of neural network structure search to obtain a round of image classification network. To further improve the performance of the selected network, multiple rounds of neural network structure search can be performed to obtain the final image classification network. Exemplarily, the image classification network in steps 1 to 4 is the first round of image classification network. After step 4 and before executing step 5, steps 6 to 7 below can also be executed.

[0202] In step 6, in response to the i-1th round image classification network not meeting the requirement conditions, the computer device determines an i-th round reference network and an i-th round random network in the super network, where the i-th round reference network includes a first group of i-th round operators predetermined in the super network, and the i-th round random network includes a second group of i-th round operators randomly sampled in the super network, where i is an integer greater than 1 and not greater than n.

[0203] In the seventh step, the computer device uses the training image sample set to train and update the network parameters of the first group of operators in the i-th round and the second group of operators in the i-th round.

[0204] In the eighth step, the computer device uses the verification image sample set to train and update the structural parameters of at least two candidate operators in the super network.

[0205] In the ninth step, the computer device searches the hypernetwork according to the structural parameters to obtain the i-th round image classification network, where the i-th round image classification network includes the i-th round third group of operators in the hypernetwork.

[0206] In the tenth step, the computer device repeatedly executes the above steps 6 to 9 until the n-th round image classification network meets the required conditions, and outputs the n-th round image classification network as the final image classification network, where n is a positive integer.

[0207] Exemplarily, the requirement condition may be that the accuracy of image classification performed by the image classification network is higher than a first threshold.

[0208] After obtaining the final image classification network, the fifth step can be replaced by: the computer device calls the final image classification network to perform image classification on the input image to obtain an image classification result.

[0209] For example, an exemplary embodiment of applying the image processing method provided by the present application to a target detection scenario is given.

[0210] In the first step, a computer device determines a reference network and a random network in a hypernetwork, where the hypernetwork includes at least two candidate operators, the reference network includes a first set of operators predetermined in the hypernetwork, and the random network includes a second set of operators randomly sampled in the hypernetwork, and the hypernetwork is used for target detection.

[0211] The computer device receives first indication information, where the first indication information is used to indicate a first group of operators constituting a reference network; the computer device determines the reference network in the supernetwork based on the first indication information. Exemplarily, the supernetwork includes at least one network layer, and each network layer includes at least one candidate operator. Based on the first indication information, the computer device selects a candidate operator from each network layer of the supernetwork to obtain a first group of operators to form the reference network. Exemplarily, the input data of each layer in the supernetwork is the output data of the previous layer, the input data of the first network layer is the input image, and the output data of the last network layer is the image processing result, that is, the target detection result.

[0212] Exemplarily, the reference network can be a network determined based on the developer's experience. For example, a first group of operators is determined from a super network based on the network structure of an existing target detection network with better performance. The network structure composed of the first group of operators is the same as the network structure of the existing target detection network.

[0213] Computer devices randomly sample with equal probability in the super network to obtain a second group of operators to form a random network; or, based on the structural parameters of at least two candidate operators, randomly sample in the super network to obtain a second group of operators to form a random network.

[0214] Exemplarily, the computer device calls a sampling function to calculate the one-hot vector of each network layer based on the structural parameters of at least one candidate operator of each network layer; according to the one-hot vector of each network layer, a candidate operator is selected from each network layer to obtain a second group of operators to form a random network.

[0215] Exemplarily, before executing the first step, the computer device will also initialize the structural parameters of the super network and at least two candidate operators; use the preheated image sample set to train the network parameters of at least two candidate operators in the super network, so that each operator in the super network has preliminary target detection capabilities.

[0216] In the second step, the computer device uses the training image sample set to train and update the network parameters of the first group of operators and the second group of operators.

[0217] Exemplarily, the training image sample set includes sample images and true values corresponding to the sample images, where the true values are target detection results of the sample images. For example, the true values can be position information of a target detection box that frames the target in the sample image, such as the center point coordinates, length, and width of the target detection box.

[0218] The computer device uses a training image sample set to train the first group of operators and the second group of operators respectively, so that the reference network composed of the first group of operators can output relatively accurate target detection results based on the input image; and the random network composed of the second group of operators can output relatively accurate target detection results based on the input image.

[0219] For example, the computer device inputs the first sample in the training image sample set into the random network to obtain a first predicted target detection result; updates the network parameters of the second group of operators based on the loss training between the first predicted target detection result and the first true target detection result, and the first true target detection result is the true target detection result corresponding to the first sample; inputs the second sample in the training image sample set into the reference network to obtain a second predicted target detection result; updates the network parameters of the first group of operators based on the loss training between the second predicted target detection result and the second true target detection result, and the second true target detection result is the true target detection result corresponding to the second sample.

[0220] In the third step, the computer device uses the verification image sample set to train and update the structural parameters of at least two candidate operators in the super network, and the structural parameters are used to determine the target detection network from the super network.

[0221] The computer device inputs the third sample in the verification image sample set into the hypernetwork to obtain a third predicted target detection result; and updates the structural parameters of at least two candidate operators in the hypernetwork based on the loss training of the third predicted target detection result and the third true target detection result, where the third true target detection result is the true target detection result corresponding to the third sample.

[0222] Exemplarily, the super network includes m network layers, the first network layer includes q candidate operators, and the jth network layer includes k candidate operators, where m, q, and k are all positive integers, and j is an integer greater than 1. The computer device inputs the third sample in the verification image sample set into the q candidate operators in the first network layer, respectively, to obtain q intermediate outputs; uses the structural parameters corresponding to the q candidate operators as weights, calculates the weighted sum of the q intermediate outputs, and obtains the first output of the first network layer; inputs the j-1th output of the j-1th network layer into the k candidate operators in the jth network layer, respectively, to obtain k intermediate outputs; uses the structural parameters corresponding to the k candidate operators as weights, calculates the weighted sum of the k intermediate outputs, and obtains the jth output of the jth network layer; repeats this step, and determines the mth output of the mth network layer as the third predicted target detection result.

[0223] In the fourth step, the computer device searches the hypernetwork according to the structural parameters to obtain a target detection network, where the target detection network includes the third group of operators in the hypernetwork.

[0224] In the fifth step, the computer device calls the target detection network to perform target detection on the input image and obtain the target detection result.

[0225] Exemplarily, executing steps 1 to 4 is to perform a round of neural network structure search to obtain a round of target detection network. To further improve the performance of the selected network, multiple rounds of neural network structure search can be performed to obtain the final target detection network. Exemplarily, the target detection network in steps 1 to 4 is the first round target detection network. After step 4 and before executing step 5, steps 6 to 7 below can also be executed.

[0226] In step 6, in response to the i-1th round target detection network not meeting the requirement conditions, the computer device determines an i-th round reference network and an i-th round random network in the super network, where the i-th round reference network includes a first group of i-th round operators predetermined in the super network, and the i-th round random network includes a second group of i-th round operators randomly sampled in the super network, where i is an integer greater than 1 and less than n.

[0227] In the seventh step, the computer device uses the training image sample set to train and update the network parameters of the first group of operators in the i-th round and the second group of operators in the i-th round.

[0228] In the eighth step, the computer device uses the verification image sample set to train and update the structural parameters of at least two candidate operators in the super network.

[0229] In the ninth step, the computer device searches the hypernetwork according to the structural parameters to obtain the i-th round target detection network, where the i-th round target detection network includes the i-th round third group of operators in the hypernetwork.

[0230] In the tenth step, the computer device repeatedly executes the above steps 6 to 9 until the n-th round target detection network meets the required conditions, and outputs the n-th round target detection network as the final target detection network, where n is a positive integer.

[0231] Exemplarily, the requirement condition may be that the error of target detection performed by the target detection network is lower than a second threshold.

[0232] After obtaining the final target detection network, the fifth step can be replaced by: the computer device calls the final target detection network to perform target detection on the input image to obtain the target detection result.

[0233] For example, an exemplary embodiment of applying the image processing method provided by the present application to an image segmentation scenario is given.

[0234] In the first step, a computer device determines a reference network and a random network in a hypernetwork, where the hypernetwork includes at least two candidate operators, the reference network includes a first set of operators predetermined in the hypernetwork, and the random network includes a second set of operators randomly sampled in the hypernetwork, and the hypernetwork is used for image segmentation.

[0235] The computer device receives first indication information, where the first indication information is used to indicate a first group of operators constituting a reference network; the computer device determines a reference network in a supernetwork based on the first indication information. Exemplarily, the supernetwork includes at least one network layer, each network layer includes at least one candidate operator, and the computer device selects a candidate operator from each network layer of the supernetwork based on the first indication information to obtain a first group of operators to form a reference network. Exemplarily, the input data of each layer in the supernetwork is the output data of the previous layer, the input data of the first network layer is the input image, and the output data of the last network layer is the image processing result, i.e., the image segmentation result.

[0236] Exemplarily, the reference network can be a network determined based on the developer's experience. For example, a first group of operators is determined from a super network based on the network structure of an existing image segmentation network with better performance. The network structure composed of the first group of operators is the same as the network structure of the existing image segmentation network.

[0237] Computer devices randomly sample with equal probability in the super network to obtain a second group of operators to form a random network; or, based on the structural parameters of at least two candidate operators, randomly sample in the super network to obtain a second group of operators to form a random network.

[0238] Exemplarily, the computer device calls a sampling function to calculate the one-hot vector of each network layer based on the structural parameters of at least one candidate operator of each network layer; according to the one-hot vector of each network layer, a candidate operator is selected from each network layer to obtain a second group of operators to form a random network.

[0239] Exemplarily, before executing the first step, the computer device will also initialize the structural parameters of the super network and at least two candidate operators; use the preheated image sample set to train the network parameters of at least two candidate operators in the super network, so that each operator in the super network has preliminary image segmentation capabilities.

[0240] In the second step, the computer device uses the training image sample set to train and update the network parameters of the first group of operators and the second group of operators.

[0241] Exemplarily, the training image sample set includes sample images and true values corresponding to the sample images. The true values are image segmentation results of the sample images. For example, the true values may be classification probability values to which each pixel in the sample image belongs.

[0242] The computer device uses a training image sample set to train the first group of operators and the second group of operators respectively, so that the reference network composed of the first group of operators can output a relatively accurate image segmentation result based on the input image; and the random network composed of the second group of operators can output a relatively accurate image segmentation result based on the input image.

[0243] For example, the computer device inputs the first sample in the training image sample set into the random network to obtain a first predicted image segmentation result; updates the network parameters of the second group of operators based on the loss training between the first predicted image segmentation result and the first real image segmentation result, and the first real image segmentation result is the real image segmentation result corresponding to the first sample; inputs the second sample in the training image sample set into the reference network to obtain a second predicted image segmentation result; updates the network parameters of the first group of operators based on the loss training between the second predicted image segmentation result and the second real image segmentation result, and the second real image segmentation result is the real image segmentation result corresponding to the second sample.

[0244] In the third step, the computer device uses the verification image sample set to train and update the structural parameters of at least two candidate operators in the super network, and the structural parameters are used to determine the image segmentation network from the super network.

[0245] The computer device inputs the third sample in the verification image sample set into the hypernetwork to obtain a third predicted image segmentation result; and updates the structural parameters of at least two candidate operators in the hypernetwork based on the loss training of the third predicted image segmentation result and the third real image segmentation result, where the third real image segmentation result is the real image segmentation result corresponding to the third sample.

[0246] Exemplarily, the super network includes m network layers, the first network layer includes q candidate operators, and the jth network layer includes k candidate operators, where m, q, and k are all positive integers, and j is an integer greater than 1. The computer device inputs the third sample in the verification image sample set into the q candidate operators in the first network layer, respectively, to obtain q intermediate outputs; uses the structural parameters corresponding to the q candidate operators as weights, calculates the weighted sum of the q intermediate outputs, and obtains the first output of the first network layer; inputs the j-1th output of the j-1th network layer into the k candidate operators in the jth network layer, respectively, to obtain k intermediate outputs; uses the structural parameters corresponding to the k candidate operators as weights, calculates the weighted sum of the k intermediate outputs, and obtains the jth output of the jth network layer; repeats this step, and determines the mth output of the mth network layer as the third predicted image segmentation result.

[0247] In the fourth step, the computer device searches the hypernetwork according to the structural parameters to obtain an image segmentation network, where the image segmentation network includes the third group of operators in the hypernetwork.

[0248] In the fifth step, the computer device calls the image segmentation network to perform image segmentation on the input image to obtain the image segmentation result.

[0249] Exemplarily, executing steps 1 to 4 is to perform a round of neural network structure search to obtain a round of image segmentation network. To further improve the performance of the selected network, multiple rounds of neural network structure search can be performed to obtain the final image segmentation network. Exemplarily, the image segmentation network in steps 1 to 4 is the first round of image segmentation network. After step 4 and before executing step 5, steps 6 to 7 below can also be executed.

[0250] In the sixth step, in response to the i-1th round image segmentation network not meeting the requirement conditions, the computer device determines an i-th round reference network and an i-th round random network in the super network, where the i-th round reference network includes a first group of i-th round operators predetermined in the super network, and the i-th round random network includes a second group of i-th round operators randomly sampled in the super network, where i is an integer greater than 1 and not greater than n.

[0251] In the seventh step, the computer device uses the training image sample set to train and update the network parameters of the first group of operators in the i-th round and the second group of operators in the i-th round.

[0252] In the eighth step, the computer device uses the verification image sample set to train and update the structural parameters of at least two candidate operators in the super network.

[0253] In the ninth step, the computer device searches the hypernetwork according to the structural parameters to obtain the i-th round image segmentation network, where the i-th round image segmentation network includes the i-th round third group of operators in the hypernetwork.

[0254] In the tenth step, the computer device repeatedly executes the above steps 6 to 9 until the n-th round image segmentation network meets the required conditions, and outputs the n-th round image segmentation network as the final image segmentation network, where n is a positive integer.

[0255] Exemplarily, the requirement condition may be that the accuracy of image segmentation performed by the image segmentation network is higher than a third threshold.

[0256] After obtaining the final image segmentation network, the fifth step mentioned above can be replaced by: the computer device calls the final image segmentation network to perform image segmentation on the input image to obtain an image segmentation result.

[0257] The following is an embodiment of the device of the present application. For details not described in detail in the embodiment of the device, reference can be made to the corresponding records in the above method embodiment, and no further details will be given herein.

[0258] Figure 6 The following is a schematic diagram of the structure of an image processing device provided by an exemplary embodiment of the present application. The device can be implemented as all or part of a computer device through software, hardware, or a combination of both. The device includes:

[0259] Determination module 601, configured to determine a reference network and a random network in a hypernetwork, wherein the hypernetwork includes at least two candidate operators, the reference network includes a first set of operators predetermined in the hypernetwork, and the random network includes a second set of operators randomly sampled in the hypernetwork, wherein the hypernetwork is used for image processing;

[0260] A network training module 602 is configured to train and update network parameters of the first group of operators and the second group of operators using a training image sample set;

[0261] A structure training module 603 is configured to use a validation image sample set to train and update structure parameters of the at least two candidate operators in the super network, wherein the structure parameters are used to determine an image processing network from the super network;

[0262] A search module 604 is configured to search the hypernetwork for the image processing network according to the structural parameters, wherein the image processing network includes a third group of operators in the hypernetwork;

[0263] The calling module 610 is used to call the image processing network to perform image processing on the input image to obtain an image processing result.

[0264] In an optional embodiment, the device further includes:

[0265] A receiving module 605 is configured to receive first indication information, where the first indication information is used to indicate the first group of operators constituting the reference network;

[0266] The determining module 601 is configured to determine the reference network in the supernetwork according to the first indication information.

[0267] In an optional embodiment, the super network includes at least one network layer, and each network layer includes at least one candidate operator;

[0268] The determining module 601 is configured to select a candidate operator from each network layer of the supernetwork according to the first indication information to obtain the first group of operators to form the reference network.

[0269] In an optional embodiment, the determining module 601 is configured to obtain the second group of operators by random sampling with equal probability in the super network to form the random network;

[0270] or,

[0271] The determining module 601 is configured to randomly sample the second group of operators in the super network according to the structural parameters of the at least two candidate operators to form the random network.

[0272] In an optional embodiment, the super network includes at least one network layer, each network layer includes at least one candidate operator; the apparatus further includes:

[0273] A calculation module 606 is configured to call a sampling function to calculate a one-hot vector of each network layer based on the structural parameters of the at least one candidate operator of each network layer;

[0274] The determining module 601 is configured to select a candidate operator from each network layer according to the one-hot vector of each network layer to obtain the second group of operators to form the random network.

[0275] In an optional embodiment, the image processing network is a first-round image processing network; and the apparatus further includes:

[0276] The determining module 601 is configured to determine, in response to the image processing network of the (i-1)th round not meeting the requirement condition, an i-th round reference network and an i-th round random network in the super network, wherein the i-th round reference network includes a first group of operators of the i-th round predetermined in the super network, and the i-th round random network includes a second group of operators of the i-th round randomly sampled in the super network, where i is an integer greater than 1 and less than n;

[0277] The network training module 602 is configured to use the training image sample set to train and update the network parameters of the first group of operators in the i-th round and the second group of operators in the i-th round;

[0278] The structure training module 603 is configured to use the verification image sample set to train and update the structure parameters of the at least two candidate operators in the super network;

[0279] The search module 604 is configured to search the super network according to the structural parameters to obtain an i-th round image processing network, wherein the i-th round image processing network includes the i-th round third group of operators in the super network;

[0280] A loop module 607 is configured to repeatedly execute the above steps until the n-th round of image processing network meets the requirement, and output the n-th round of image processing network as a final image processing network, where n is a positive integer;

[0281] The calling module 610 is used to call the final image processing network to perform image processing on the input image to obtain the image processing result.

[0282] In an optional embodiment, the reference network is a first-round reference network; and the apparatus further includes:

[0283] A calculation module 606 is configured to calculate performance parameters of the image processing network of the (i-1)th round and the reference network of the (i-1)th round, where i is an integer greater than 1 and less than n;

[0284] The determining module 601 is configured to determine the network with better performance parameters between the (i-1)th round image processing network and the (i-1)th round reference network as the (i-1)th round reference network.

[0285] In an optional embodiment, the device further includes:

[0286] A receiving module 605 is configured to receive second indication information, where the second indication information is used to indicate the first group of operators in the i-th round constituting the i-th round reference network;

[0287] The determining module 601 is configured to determine the i-th round reference network in the super network according to the second indication information.

[0288] In an optional embodiment, the network training module 602 is configured to input a first sample in the training image sample set into the random network to obtain a first prediction value;

[0289] The network training module 602 is configured to update the network parameters of the second group of operators based on the loss between the first predicted value and a first true value, where the first true value is a true value corresponding to the first sample;

[0290] The network training module 602 is configured to input a second sample in the training image sample set into the reference network to obtain a second prediction value;

[0291] The network training module 602 is used to update the network parameters of the first group of operators according to the loss training of the second predicted value and the second true value, where the second true value is the true value corresponding to the second sample.

[0292] In an optional embodiment, the structure training module 603 is configured to input a third sample in the verification image sample set into the hypernetwork to obtain a third prediction value;

[0293] The structure training module 603 is used to update the structure parameters of the at least two candidate operators in the super network according to the loss training of the third predicted value and the third true value, where the third true value is the true value corresponding to the third sample.

[0294] In an optional embodiment, the super network includes m network layers, the first network layer includes q candidate operators, the jth network layer includes k candidate operators, m, q, k are positive integers, and j is an integer greater than 1;

[0295] The structure training module 603 is configured to input the third sample in the verification image sample set into the q candidate operators in the first network layer to obtain q intermediate outputs; use the structure parameters corresponding to the q candidate operators as weights, calculate the weighted sum of the q intermediate outputs, and obtain the first output of the first network layer;

[0296] The structure training module 603 is used to input the j-1th output of the j-1th network layer into the k candidate operators in the jth network layer respectively to obtain k intermediate outputs; use the structure parameters corresponding to the k candidate operators as weights, calculate the weighted sum of the k intermediate outputs, and obtain the jth output of the jth network layer; repeat this step to determine the mth output of the mth network layer as the third prediction value.

[0297] In an optional embodiment, the device further includes:

[0298] An initialization module 608 is configured to initialize the structural parameters of the supernetwork and the at least two candidate operators;

[0299] The preheating module 609 is configured to train the network parameters of the at least two candidate operators in the super network using a preheating image sample set.

[0300] In an optional embodiment, the preheating module 609 is configured to determine a preheating network in the super network, where the preheating network includes a fourth group of operators randomly sampled from the super network;

[0301] The preheating module 609 is configured to use the preheating image sample set to train and update the network parameters of the fourth group of operators;

[0302] The preheating module 609 is configured to repeat the above steps to train the network parameters of the at least two candidate operators in the super network.

[0303] Determination module 601, configured to determine a reference network and a random network in a supernetwork, wherein the supernetwork includes at least two candidate operators, the reference network includes a first set of operators predetermined in the supernetwork, and the random network includes a second set of operators randomly sampled in the supernetwork, wherein the supernetwork is used for image classification;

[0304] A network training module 602 is configured to train and update network parameters of the first group of operators and the second group of operators using a training image sample set;

[0305] A structure training module 603 is configured to use a validation image sample set to train and update structure parameters of the at least two candidate operators in the super network, wherein the structure parameters are used to determine an image classification network from the super network;

[0306] A search module 604 is configured to search the hypernetwork for the image classification network according to the structural parameters, wherein the image classification network includes a third group of operators in the hypernetwork;

[0307] The calling module 610 is used to call the image classification network to perform image classification on the input image to obtain an image classification result.

[0308] In an optional embodiment, the device further includes:

[0309] A receiving module 605 is configured to receive first indication information, where the first indication information is used to indicate the first group of operators constituting the reference network;

[0310] The determining module 601 is configured to determine the reference network in the supernetwork according to the first indication information.

[0311] In an optional embodiment, the super network includes at least one network layer, and each network layer includes at least one candidate operator;

[0312] The determining module 601 is configured to select a candidate operator from each network layer of the supernetwork according to the first indication information to obtain the first group of operators to form the reference network.

[0313] In an optional embodiment, the determining module 601 is configured to obtain the second group of operators by random sampling with equal probability in the super network to form the random network;

[0314] or,

[0315] The determining module 601 is configured to randomly sample the second group of operators in the super network according to the structural parameters of the at least two candidate operators to form the random network.

[0316] In an optional embodiment, the super network includes at least one network layer, each network layer includes at least one candidate operator; the apparatus further includes:

[0317] A calculation module 606 is configured to call a sampling function to calculate a one-hot vector of each network layer based on the structural parameters of the at least one candidate operator of each network layer;

[0318] The determining module 601 is configured to select a candidate operator from each network layer according to the one-hot vector of each network layer to obtain the second group of operators to form the random network.

[0319] In an optional embodiment, the image classification network is a first-round image classification network; and the apparatus further includes:

[0320] The determining module 601 is configured to determine, in response to the i-1th round image classification network not meeting the requirement condition, an i-th round reference network and an i-th round random network in the supernetwork, wherein the i-th round reference network includes a first group of i-th round operators predetermined in the supernetwork, and the i-th round random network includes a second group of i-th round operators randomly sampled in the supernetwork, where i is an integer greater than 1 and less than n;

[0321] The network training module 602 is configured to use the training image sample set to train and update the network parameters of the first group of operators in the i-th round and the second group of operators in the i-th round;

[0322] The structure training module 603 is configured to use the verification image sample set to train and update the structure parameters of the at least two candidate operators in the super network;

[0323] The search module 604 is configured to search the super network according to the structural parameters to obtain an i-th round image classification network, wherein the i-th round image classification network includes the i-th round third group of operators in the super network;

[0324] A loop module 607 is configured to repeatedly execute the above steps until the n-th round image classification network meets the requirement, and output the n-th round image classification network as a final image classification network, where n is a positive integer;

[0325] The calling module 610 is used to call the final image classification network to perform image classification on the input image to obtain the image classification result.

[0326] In an optional embodiment, the reference network is a first-round reference network; and the apparatus further includes:

[0327] A calculation module 606 is configured to calculate the performance parameters of the i-1th round image classification network and the i-1th round reference network, where i is an integer greater than 1 and less than n;

[0328] The determination module 601 is configured to determine the network with better performance parameters between the i-1th round image classification network and the i-1th round reference network as the i-th round reference network.

[0329] In an optional embodiment, the device further includes:

[0330] A receiving module 605 is configured to receive second indication information, where the second indication information is used to indicate the first group of operators in the i-th round constituting the i-th round reference network;

[0331] The determining module 601 is configured to determine the i-th round reference network in the super network according to the second indication information.

[0332] In an optional embodiment, the network training module 602 is configured to input a first sample in the training image sample set into the random network to obtain a first prediction classification result;

[0333] The network training module 602 is configured to update the network parameters of the second group of operators based on the loss training between the first predicted classification result and a first true classification result, where the first true classification result is the true classification result corresponding to the first sample;

[0334] The network training module 602 is configured to input a second sample in the training image sample set into the reference network to obtain a second prediction classification result;

[0335] The network training module 602 is used to update the network parameters of the first group of operators based on the loss training of the second predicted classification result and the second true classification result, where the second true classification result is the true classification result corresponding to the second sample.

[0336] In an optional embodiment, the structure training module 603 is configured to input a third sample in the verification image sample set into the hypernetwork to obtain a third prediction classification result;

[0337] The structure training module 603 is used to update the structure parameters of the at least two candidate operators in the super network based on the loss training of the third predicted classification result and the third true classification result, where the third true classification result is the true classification result corresponding to the third sample.

[0338] In an optional embodiment, the super network includes m network layers, the first network layer includes q candidate operators, the jth network layer includes k candidate operators, m, q, k are positive integers, and j is an integer greater than 1;

[0339] The structure training module 603 is configured to input the third sample in the verification image sample set into the q candidate operators in the first network layer to obtain q intermediate outputs; use the structure parameters corresponding to the q candidate operators as weights, calculate the weighted sum of the q intermediate outputs, and obtain the first output of the first network layer;

[0340] The structure training module 603 is used to input the j-1th output of the j-1th network layer into the k candidate operators in the jth network layer respectively to obtain k intermediate outputs; use the structure parameters corresponding to the k candidate operators as weights, calculate the weighted sum of the k intermediate outputs, and obtain the jth output of the jth network layer; repeat this step to determine the mth output of the mth network layer as the third prediction classification result.

[0341] In an optional embodiment, the device further includes:

[0342] An initialization module 608 is configured to initialize the structural parameters of the supernetwork and the at least two candidate operators;

[0343] The preheating module 609 is configured to train the network parameters of the at least two candidate operators in the super network using a preheating image sample set.

[0344] In an optional embodiment, the preheating module 609 is configured to determine a preheating network in the super network, where the preheating network includes a fourth group of operators randomly sampled from the super network;

[0345] The preheating module 609 is configured to use the preheating image sample set to train and update the network parameters of the fourth group of operators;

[0346] The preheating module 609 is configured to repeat the above steps to train the network parameters of the at least two candidate operators in the super network.

[0347] In an optional embodiment, a determination module 601 is configured to determine a reference network and a random network in a supernetwork, wherein the supernetwork includes at least two candidate operators, the reference network includes a first group of operators predetermined in the supernetwork, and the random network includes a second group of operators randomly sampled in the supernetwork, and the supernetwork is used for target detection;

[0348] A network training module 602 is configured to train and update network parameters of the first group of operators and the second group of operators using a training image sample set;

[0349] A structure training module 603 is configured to use a validation image sample set to train and update structure parameters of the at least two candidate operators in the super network, wherein the structure parameters are used to determine an object detection network from the super network;

[0350] A search module 604 is configured to search the hypernetwork for the target detection network according to the structural parameters, wherein the target detection network includes a third group of operators in the hypernetwork;

[0351] The calling module 610 is used to call the target detection network to perform target detection on the input image to obtain a target detection result.

[0352] In an optional embodiment, the device further includes:

[0353] A receiving module 605 is configured to receive first indication information, where the first indication information is used to indicate the first group of operators constituting the reference network;

[0354] The determining module 601 is configured to determine the reference network in the supernetwork according to the first indication information.

[0355] In an optional embodiment, the super network includes at least one network layer, and each network layer includes at least one candidate operator;

[0356] The determining module 601 is configured to select a candidate operator from each network layer of the supernetwork according to the first indication information to obtain the first group of operators to form the reference network.

[0357] In an optional embodiment, the determining module 601 is configured to obtain the second group of operators by random sampling with equal probability in the super network to form the random network;

[0358] or,

[0359] The determining module 601 is configured to randomly sample the second group of operators in the super network according to the structural parameters of the at least two candidate operators to form the random network.

[0360] In an optional embodiment, the super network includes at least one network layer, each network layer includes at least one candidate operator; the apparatus further includes:

[0361] A calculation module 606 is configured to call a sampling function to calculate a one-hot vector of each network layer based on the structural parameters of the at least one candidate operator of each network layer;

[0362] The determining module 601 is configured to select a candidate operator from each network layer according to the one-hot vector of each network layer to obtain the second group of operators to form the random network.

[0363] In an optional embodiment, the target detection network is a first-round target detection network; and the apparatus further includes:

[0364] The determining module 601 is configured to determine, in response to the i-1th round object detection network not meeting the requirement condition, an i-th round reference network and an i-th round random network in the supernetwork, wherein the i-th round reference network includes a first group of i-th round operators predetermined in the supernetwork, and the i-th round random network includes a second group of i-th round operators randomly sampled in the supernetwork, where i is an integer greater than 1 and less than n;

[0365] The network training module 602 is configured to use the training image sample set to train and update the network parameters of the first group of operators in the i-th round and the second group of operators in the i-th round;

[0366] The structure training module 603 is configured to use the verification image sample set to train and update the structure parameters of the at least two candidate operators in the super network;

[0367] The search module 604 is configured to search the super network according to the structural parameters to obtain an i-th round target detection network, wherein the i-th round target detection network includes the i-th round third group of operators in the super network;

[0368] A loop module 607 is configured to repeatedly execute the above steps until the target detection network of the nth round meets the requirement, and output the target detection network of the nth round as a final target detection network, where n is a positive integer;

[0369] The calling module 610 is used to call the final object detection network to perform object detection on the input image to obtain the object detection result.

[0370] In an optional embodiment, the reference network is a first-round reference network; and the apparatus further includes:

[0371] A calculation module 606 is configured to calculate performance parameters of the i-1th round target detection network and the i-1th round reference network, where i is an integer greater than 1 and less than n;

[0372] The determining module 601 is configured to determine the network with better performance parameters between the (i-1)th round target detection network and the (i-1)th round reference network as the (i-1)th round reference network.

[0373] In an optional embodiment, the device further includes:

[0374] A receiving module 605 is configured to receive second indication information, where the second indication information is used to indicate the first group of operators in the i-th round constituting the i-th round reference network;

[0375] The determining module 601 is configured to determine the i-th round reference network in the super network according to the second indication information.

[0376] In an optional embodiment, the network training module 602 is configured to input a first sample in the training image sample set into the random network to obtain a first predicted target detection result;

[0377] The network training module 602 is configured to update the network parameters of the second group of operators based on the loss training between the first predicted target detection result and a first true target detection result, where the first true target detection result is the true target detection result corresponding to the first sample;

[0378] The network training module 602 is configured to input a second sample in the training image sample set into the reference network to obtain a second predicted target detection result;

[0379] The network training module 602 is used to update the network parameters of the first group of operators based on the loss training of the second predicted target detection result and the second true target detection result, where the second true target detection result is the true target detection result corresponding to the second sample.

[0380] In an optional embodiment, the structure training module 603 is configured to input a third sample in the verification image sample set into the hypernetwork to obtain a third predicted target detection result;

[0381] The structure training module 603 is used to update the structure parameters of the at least two candidate operators in the super network based on the loss training of the third predicted target detection result and the third true target detection result, where the third true target detection result is the true target detection result corresponding to the third sample.

[0382] In an optional embodiment, the super network includes m network layers, the first network layer includes q candidate operators, the jth network layer includes k candidate operators, m, q, k are positive integers, and j is an integer greater than 1;

[0383] The structure training module 603 is configured to input the third sample in the verification image sample set into the q candidate operators in the first network layer to obtain q intermediate outputs; use the structure parameters corresponding to the q candidate operators as weights, calculate the weighted sum of the q intermediate outputs, and obtain the first output of the first network layer;

[0384] The structure training module 603 is used to input the j-1th output of the j-1th network layer into the k candidate operators in the jth network layer respectively to obtain k intermediate outputs; use the structure parameters corresponding to the k candidate operators as weights, calculate the weighted sum of the k intermediate outputs, and obtain the jth output of the jth network layer; repeat this step to determine the mth output of the mth network layer as the third predicted target detection result.

[0385] In an optional embodiment, the device further includes:

[0386] An initialization module 608 is configured to initialize the structural parameters of the supernetwork and the at least two candidate operators;

[0387] The preheating module 609 is configured to train the network parameters of the at least two candidate operators in the super network using a preheating image sample set.

[0388] In an optional embodiment, the preheating module 609 is configured to determine a preheating network in the super network, where the preheating network includes a fourth group of operators randomly sampled from the super network;

[0389] The preheating module 609 is configured to use the preheating image sample set to train and update the network parameters of the fourth group of operators;

[0390] The preheating module 609 is configured to repeat the above steps to train the network parameters of the at least two candidate operators in the super network.

[0391] In an optional embodiment, the determination module 601 is configured to determine a reference network and a random network in a supernetwork, the supernetwork including at least two candidate operators, the reference network including a first group of operators predetermined in the supernetwork, the random network including a second group of operators randomly sampled in the supernetwork, and the supernetwork being used for image segmentation;

[0392] A network training module 602 is configured to train and update network parameters of the first group of operators and the second group of operators using a training image sample set;

[0393] A structure training module 603 is configured to use a validation image sample set to train and update structure parameters of the at least two candidate operators in the super network, wherein the structure parameters are used to determine an image segmentation network from the super network;

[0394] A search module 604 is configured to search the hypernetwork for the image segmentation network according to the structural parameters, wherein the image segmentation network includes a third group of operators in the hypernetwork;

[0395] The calling module 610 is used to call the image segmentation network to perform image segmentation on the input image to obtain an image segmentation result.

[0396] In an optional embodiment, the device further includes:

[0397] A receiving module 605 is configured to receive first indication information, where the first indication information is used to indicate the first group of operators constituting the reference network;

[0398] The determining module 601 is configured to determine the reference network in the supernetwork according to the first indication information.

[0399] In an optional embodiment, the super network includes at least one network layer, and each network layer includes at least one candidate operator;

[0400] The determining module 601 is configured to select a candidate operator from each network layer of the supernetwork according to the first indication information to obtain the first group of operators to form the reference network.

[0401] In an optional embodiment, the determining module 601 is configured to obtain the second group of operators by random sampling with equal probability in the super network to form the random network;

[0402] or,

[0403] The determining module 601 is configured to randomly sample the second group of operators in the super network according to the structural parameters of the at least two candidate operators to form the random network.

[0404] In an optional embodiment, the super network includes at least one network layer, each network layer includes at least one candidate operator; the apparatus further includes:

[0405] A calculation module 606 is configured to call a sampling function to calculate a one-hot vector of each network layer based on the structural parameters of the at least one candidate operator of each network layer;

[0406] The determining module 601 is configured to select a candidate operator from each network layer according to the one-hot vector of each network layer to obtain the second group of operators to form the random network.

[0407] In an optional embodiment, the image segmentation network is a first-round image segmentation network; and the apparatus further includes:

[0408] The determining module 601 is configured to determine, in response to the i-1th round image segmentation network not meeting the requirement condition, an i-th round reference network and an i-th round random network in the supernetwork, wherein the i-th round reference network includes a first group of i-th round operators predetermined in the supernetwork, and the i-th round random network includes a second group of i-th round operators randomly sampled in the supernetwork, where i is an integer greater than 1 and less than n;

[0409] The network training module 602 is configured to use the training image sample set to train and update the network parameters of the first group of operators in the i-th round and the second group of operators in the i-th round;

[0410] The structure training module 603 is configured to use the verification image sample set to train and update the structure parameters of the at least two candidate operators in the super network;

[0411] The search module 604 is configured to search the super network according to the structural parameters to obtain an i-th round image segmentation network, wherein the i-th round image segmentation network includes the i-th round third group of operators in the super network;

[0412] A loop module 607 is configured to repeatedly execute the above steps until the n-th round image segmentation network meets the requirement, and output the n-th round image segmentation network as a final image segmentation network, where n is a positive integer;

[0413] The calling module 610 is used to call the final image segmentation network to perform image segmentation on the input image to obtain the image segmentation result.

[0414] In an optional embodiment, the reference network is a first-round reference network; and the apparatus further includes:

[0415] A calculation module 606 is used to calculate the performance parameters of the i-1th round image segmentation network and the i-1th round reference network, where i is an integer greater than 1 and less than n;

[0416] The determination module 601 is configured to determine the network with better performance parameters between the i-1th round image segmentation network and the i-1th round reference network as the i-th round reference network.

[0417] In an optional embodiment, the device further includes:

[0418] A receiving module 605 is configured to receive second indication information, where the second indication information is used to indicate the first group of operators in the i-th round constituting the i-th round reference network;

[0419] The determining module 601 is configured to determine the i-th round reference network in the super network according to the second indication information.

[0420] In an optional embodiment, the network training module 602 is configured to input a first sample in the training image sample set into the random network to obtain a first predicted image segmentation result;

[0421] The network training module 602 is configured to update the network parameters of the second group of operators based on loss training between the first predicted image segmentation result and a first real image segmentation result, where the first real image segmentation result is a real image segmentation result corresponding to the first sample;

[0422] The network training module 602 is configured to input a second sample in the training image sample set into the reference network to obtain a second predicted image segmentation result;

[0423] The network training module 602 is used to update the network parameters of the first group of operators based on the loss training of the second predicted image segmentation result and the second real image segmentation result, where the second real image segmentation result is the real image segmentation result corresponding to the second sample.

[0424] In an optional embodiment, the structure training module 603 is configured to input a third sample in the verification image sample set into the hypernetwork to obtain a third predicted image segmentation result;

[0425] The structure training module 603 is used to update the structure parameters of the at least two candidate operators in the super network based on the loss training of the third predicted image segmentation result and the third real image segmentation result, where the third real image segmentation result is the real image segmentation result corresponding to the third sample.

[0426] In an optional embodiment, the super network includes m network layers, the first network layer includes q candidate operators, the jth network layer includes k candidate operators, m, q, k are positive integers, and j is an integer greater than 1;

[0427] The structure training module 603 is configured to input the third sample in the verification image sample set into the q candidate operators in the first network layer to obtain q intermediate outputs; use the structure parameters corresponding to the q candidate operators as weights, calculate the weighted sum of the q intermediate outputs, and obtain the first output of the first network layer;

[0428] The structure training module 603 is used to input the j-1th output of the j-1th network layer into the k candidate operators in the jth network layer respectively to obtain k intermediate outputs; use the structure parameters corresponding to the k candidate operators as weights, calculate the weighted sum of the k intermediate outputs, and obtain the jth output of the jth network layer; repeat this step to determine the mth output of the mth network layer as the third predicted image segmentation result.

[0429] In an optional embodiment, the device further includes:

[0430] An initialization module 608 is configured to initialize the structural parameters of the supernetwork and the at least two candidate operators;

[0431] The preheating module 609 is configured to train the network parameters of the at least two candidate operators in the super network using a preheating image sample set.

[0432] In an optional embodiment, the preheating module 609 is configured to determine a preheating network in the super network, where the preheating network includes a fourth group of operators randomly sampled from the super network;

[0433] The preheating module 609 is configured to use the preheating image sample set to train and update the network parameters of the fourth group of operators;

[0434] The preheating module 609 is configured to repeat the above steps to train the network parameters of the at least two candidate operators in the super network.

[0435] Figure 7Schematic diagram of the structure of a server provided by one embodiment of the present application. Specifically, server 800 includes a central processing unit (CPU) 801, a system memory 804 including random access memory (RAM) 802 and read-only memory (ROM) 803, and a system bus 805 connecting system memory 804 and CPU 801. Server 800 also includes a basic input / output system (I / O system) 806 that facilitates information transmission between various components within the computer, and a mass storage device 807 for storing an operating system 813, application programs 814, and other program modules 815.

[0436] The basic input / output system 806 includes a display 808 for displaying information and an input device 809, such as a mouse and keyboard, for user account input. Both the display 808 and the input device 809 are connected to the central processing unit 801 via an input / output controller 810 connected to the system bus 805. The basic input / output system 806 may also include an input / output controller 810 for receiving and processing input from various other devices, such as a keyboard, mouse, or electronic stylus. Similarly, the input / output controller 810 also provides output to a display screen, printer, or other types of output devices.

[0437] The mass storage device 807 is connected to the central processing unit 801 via a mass storage controller (not shown) connected to the system bus 805. The mass storage device 807 and its associated computer-readable media provide non-volatile storage for the server 800. In other words, the mass storage device 807 may include computer-readable media (not shown) such as a hard disk or a CD-ROM drive.

[0438] Without loss of generality, computer-readable media may include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented by any method or technology for storing information such as computer-readable instructions, data structures, program modules or other data. Computer storage media include RAM, ROM, Erasable Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), flash memory or other solid-state storage technologies, CD-ROM, Digital Versatile Disc (DVD) or other optical storage, tape cassettes, magnetic tape, disk storage or other magnetic storage devices. Of course, those skilled in the art will appreciate that computer storage media are not limited to the above-mentioned ones. The above-mentioned system memory 804 and mass storage device 807 can be collectively referred to as memory.

[0439] According to various embodiments of the present application, the server 800 may also be connected to a remote computer on a network such as the Internet for operation. That is, the server 800 may be connected to the network 812 via the network interface unit 811 connected to the system bus 805, or the network interface unit 811 may be used to connect to other types of networks or remote computer systems (not shown).

[0440] The present application also provides a terminal, which includes a processor and a memory, wherein the memory stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement the image processing method provided by each of the above method embodiments. It should be noted that the terminal can be as follows Figure 8 The terminal provided.

[0441] Figure 8 The following is a block diagram of a terminal 900 according to an exemplary embodiment of the present application. Terminal 900 may be a smartphone, tablet computer, MP3 player (Moving Picture Experts Group Audio Layer III), MP4 player (Moving Picture Experts Group Audio Layer IV), laptop computer, or desktop computer. Terminal 900 may also be referred to as a user account device, portable terminal, laptop terminal, desktop terminal, or other similar names.

[0442] Typically, the terminal 900 includes a processor 901 and a memory 902 .

[0443] The processor 901 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 901 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), or PLA (Programmable Logic Array). The processor 901 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 901 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 901 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0444] The memory 902 may include one or more computer-readable storage media, which may be non-transitory. The memory 902 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices and flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 902 is used to store at least one instruction, which is used to be executed by the processor 901 to implement the image processing method or image processing method provided in the method embodiment of the present application.

[0445] In some embodiments, terminal 900 may optionally include a peripheral device interface 903 and at least one peripheral device. The processor 901, memory 902, and peripheral device interface 903 may be connected via a bus or signal lines. Each peripheral device may be connected to peripheral device interface 903 via a bus, signal lines, or circuit boards. Specifically, the peripheral device may include at least one of a radio frequency circuit 904, a display screen 905, a camera assembly 906, an audio circuit 907, a positioning assembly 908, and a power supply 909.

[0446] The peripheral device interface 903 can be used to connect at least one I / O (Input / Output)-related peripheral device to the processor 901 and the memory 902. In some embodiments, the processor 901, the memory 902, and the peripheral device interface 903 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 901, the memory 902, and the peripheral device interface 903 can be implemented on separate chips or circuit boards, which is not limited in this embodiment.

[0447] The RF circuit 904 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The RF circuit 904 communicates with communication networks and other communication devices via electromagnetic signals. The RF circuit 904 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals into electrical signals. Exemplarily, the RF circuit 904 includes an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user account identity module card, and the like. The RF circuit 904 can communicate with other terminals via at least one wireless communication protocol. Such wireless communication protocols include, but are not limited to, the World Wide Web, metropolitan area networks, intranets, various generations of mobile communication networks (2G, 3G, 4G, and 5G), wireless local area networks, and / or WiFi (Wireless Fidelity) networks. In some embodiments, the RF circuit 904 may also include circuitry related to Near Field Communication (NFC), although this application does not limit this.

[0448] Display screen 905 is used to display a user interface (UI). This UI may include graphics, text, icons, videos, or any combination thereof. When display screen 905 is a touchscreen display, it is also capable of collecting touch signals on or above the surface of display screen 905. These touch signals can be input as control signals to processor 901 for processing. Display screen 905 can also be used to provide virtual buttons and / or a virtual keyboard, also known as soft buttons and / or a soft keyboard. In some embodiments, there can be a single display screen 905, located on the front panel of terminal 900. In other embodiments, there can be at least two display screens 905, located on different surfaces of terminal 900 or in a foldable design. In still other embodiments, display screen 905 can be a flexible display, located on a curved or foldable surface of terminal 900. Display screen 905 can also be configured as a non-rectangular, irregular shape, also known as a special-shaped screen. Display screen 905 can be made of materials such as LCD (Liquid Crystal Display) and OLED (Organic Light-Emitting Diode).

[0449] The camera assembly 906 is used to capture images or videos. Exemplarily, the camera assembly 906 includes a front camera and a rear camera. Typically, the front camera is arranged on the front panel of the terminal, and the rear camera is arranged on the back of the terminal. In some embodiments, there are at least two rear cameras, which are any one of a main camera, a depth of field camera, a wide-angle camera, and a telephoto camera, so as to realize the fusion of the main camera and the depth of field camera to realize the background blur function, the fusion of the main camera and the wide-angle camera to realize panoramic shooting and VR (Virtual Reality) shooting function or other fusion shooting functions. In some embodiments, the camera assembly 906 may also include a flash. The flash can be a monochrome temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm light flash and a cold light flash, which can be used for light compensation at different color temperatures.

[0450] The audio circuit 907 may include a microphone and a speaker. The microphone is used to collect sound waves from the user account and the environment, and convert the sound waves into electrical signals that are input into the processor 901 for processing, or input into the radio frequency circuit 904 to achieve voice communication. For the purpose of stereo sound collection or noise reduction, there can be multiple microphones, each located in different parts of the terminal 900. The microphone can also be an array microphone or an omnidirectional collection microphone. The speaker is used to convert electrical signals from the processor 901 or the radio frequency circuit 904 into sound waves. The speaker can be a traditional thin film speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can not only convert electrical signals into sound waves audible to humans, but also convert electrical signals into sound waves inaudible to humans for purposes such as ranging. In some embodiments, the audio circuit 907 may also include a headphone jack.

[0451] Positioning component 908 is used to locate the current geographic location of terminal 900 to implement navigation or LBS (Location Based Service). Positioning component 908 can be a positioning component based on the US GPS (Global Positioning System), China's Beidou system, or Russia's Galileo system.

[0452] Power supply 909 is used to power various components in terminal 900. Power supply 909 can be AC power, DC power, a disposable battery, or a rechargeable battery. When power supply 909 includes a rechargeable battery, the rechargeable battery can be a wired rechargeable battery or a wireless rechargeable battery. A wired rechargeable battery is a battery that is charged via a wired line, while a wireless rechargeable battery is a battery that is charged via a wireless coil. The rechargeable battery can also be used to support fast charging technology.

[0453] In some embodiments, the terminal 900 further includes one or more sensors 910 , including but not limited to: an acceleration sensor 911 , a gyroscope sensor 912 , a pressure sensor 913 , a fingerprint sensor 914 , an optical sensor 915 , and a proximity sensor 916 .

[0454] The accelerometer 911 can detect the magnitude of acceleration along the three coordinate axes of the coordinate system established by the terminal 900. For example, the accelerometer 911 can be used to detect the components of gravity acceleration along the three coordinate axes. The processor 901 can control the display screen 905 to display the user account interface in either a landscape or portrait view based on the gravity acceleration signal collected by the accelerometer 911. The accelerometer 911 can also be used to collect motion data for games or user accounts.

[0455] The gyroscope sensor 912 can detect the orientation and rotation angle of the terminal 900. It can also work with the accelerometer 911 to collect 3D motions of the user account on the terminal 900. Based on the data collected by the gyroscope sensor 912, the processor 901 can implement the following functions: motion sensing (for example, changing the UI based on the user account's tilt operation), image stabilization during shooting, game control, and inertial navigation.

[0456] The pressure sensor 913 can be set on the side frame of the terminal 900 and / or the lower layer of the display screen 905. When the pressure sensor 913 is set on the side frame of the terminal 900, it can detect the user account's grip signal on the terminal 900, and the processor 901 performs left and right hand recognition or shortcut operations based on the grip signal collected by the pressure sensor 913. When the pressure sensor 913 is set on the lower layer of the display screen 905, the processor 901 controls the operable controls on the UI interface based on the pressure operation of the user account on the display screen 905. Operable controls include at least one of button controls, scroll bar controls, icon controls, and menu controls.

[0457] The fingerprint sensor 914 is used to collect the fingerprint of the user account. The processor 901 identifies the identity of the user account based on the fingerprint collected by the fingerprint sensor 914, or the fingerprint sensor 914 identifies the identity of the user account based on the collected fingerprint. When the identity of the user account is identified as a trusted identity, the processor 901 authorizes the user account to perform relevant sensitive operations, such as unlocking the screen, viewing encrypted information, downloading software, making payments, and changing settings. The fingerprint sensor 914 can be set on the front, back, or side of the terminal 900. When a physical button or manufacturer logo is set on the terminal 900, the fingerprint sensor 914 can be integrated with the physical button or manufacturer logo.

[0458] The optical sensor 915 is used to detect ambient light intensity. In one embodiment, the processor 901 can control the display brightness of the display screen 905 based on the ambient light intensity detected by the optical sensor 915. Specifically, when the ambient light intensity is high, the display brightness of the display screen 905 is increased; when the ambient light intensity is low, the display brightness of the display screen 905 is decreased. In another embodiment, the processor 901 can also dynamically adjust the shooting parameters of the camera assembly 906 based on the ambient light intensity detected by the optical sensor 915.

[0459] Proximity sensor 916, also known as a distance sensor, is typically located on the front panel of terminal 900. Proximity sensor 916 is used to detect the distance between the user account and the front of terminal 900. In one embodiment, when proximity sensor 916 detects that the distance between the user account and the front of terminal 900 is gradually decreasing, processor 901 controls display screen 905 to switch from the screen-on state to the screen-off state. When proximity sensor 916 detects that the distance between the user account and the front of terminal 900 is gradually increasing, processor 901 controls display screen 905 to switch from the screen-off state to the screen-on state.

[0460] Those skilled in the art will understand that Figure 8 The structure shown in the figure does not constitute a limitation on the terminal 900, and the terminal 900 may include more or fewer components than shown in the figure, or combine certain components, or adopt a different component arrangement.

[0461] The memory also includes one or more programs, which are stored in the memory and include instructions for performing the image processing method provided in the embodiment of the present application.

[0462] The present application also provides a computer device, which includes: a processor and a memory, wherein the storage medium stores at least one instruction, at least one program, code set or instruction set, and the at least one instruction, at least one program, code set or instruction set is loaded and executed by the processor to implement the image processing method provided by the above-mentioned method embodiments.

[0463] The present application also provides a computer-readable storage medium, which stores at least one instruction, at least one program, code set or instruction set, and the at least one instruction, at least one program, code set or instruction set is loaded and executed by a processor to implement the image processing method provided by the above-mentioned method embodiments.

[0464] The present application also provides a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the image processing method provided in the above-mentioned optional implementation.

[0465] It should be understood that the term "plurality" used herein refers to two or more. "And / or" describes a relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can mean: A exists alone, A and B exist simultaneously, or B exists alone. The character " / " generally indicates an "or" relationship between the associated objects.

[0466] Those skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware, or by a program to instruct the relevant hardware, and the program may be stored in a computer-readable storage medium, which may be a read-only memory, a disk, or an optical disk, etc.

[0467] The above are only optional embodiments of the present application and are not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should be included in the scope of protection of the present application.

Claims

1. An image processing method, characterized in that: The method comprises: Determining a reference network and a random network in a hypernetwork, wherein the hypernetwork includes at least two candidate operators, the reference network includes a first group of operators predetermined in the hypernetwork, and the random network includes a second group of operators randomly sampled in the hypernetwork, wherein the hypernetwork is used for image processing; Using a training image sample set to train and update network parameters of the first group of operators and the second group of operators; Using a validation image sample set to train and update structural parameters of the at least two candidate operators in the super network, the structural parameters being used to determine an image processing network from the super network; Searching the image processing network from the super network according to the structural parameters, wherein the image processing network includes a third group of operators in the super network; The image processing network is called to perform image processing on the input image to obtain an image processing result.

2. The method according to claim 1, characterized in that Determining a reference network in the supernetwork includes: receiving first indication information, where the first indication information is used to indicate the first group of operators constituting the reference network; The reference network is determined in the supernetwork according to the first indication information.

3. The method according to claim 2, characterized in that The super network includes at least one network layer, and each network layer includes at least one candidate operator; The determining the reference network in the supernetwork according to the first indication information includes: According to the first indication information, a candidate operator is selected from each network layer of the super network to obtain the first group of operators to form the reference network.

4. The method according to any one of claims 1 to 3, characterized in that Determining a random network in the super network includes: The second group of operators is obtained by random sampling with equal probability in the super network to form the random network; or, According to the structural parameters of the at least two candidate operators, the second group of operators is obtained by random sampling in the super network to form the random network.

5. The method according to claim 4, characterized in that The super network includes at least one network layer, and each network layer includes at least one candidate operator; The step of randomly sampling the second group of operators in the super network based on the structural parameters of the at least two candidate operators to form the random network comprises: Calling a sampling function to calculate a one-hot vector of each network layer based on the structural parameters of the at least one candidate operator of each network layer; According to the one-hot vector of each network layer, a candidate operator is selected from each network layer to obtain the second group of operators to form the random network.

6. The method according to any one of claims 1 to 3, characterized in that: The image processing network is a first-round image processing network; the method further includes: In response to the image processing network of the (i-1)th round not meeting the requirement condition, determining an i-th round reference network and an i-th round random network in the super network, wherein the i-th round reference network includes a first group of operators of the i-th round predetermined in the super network, and the i-th round random network includes a second group of operators of the i-th round randomly sampled in the super network, where i is an integer greater than 1 and less than n; Using the training image sample set to train and update the network parameters of the first group of operators in the i-th round and the second group of operators in the i-th round; Using the verification image sample set to train and update the structural parameters of the at least two candidate operators in the super network; Searching the super network according to the structural parameters to obtain an i-th round image processing network, wherein the i-th round image processing network includes the i-th round third group of operators in the super network; Repeat the above steps until the n-th round image processing network meets the requirement, and output the n-th round image processing network as the final image processing network, where n is a positive integer; The final image processing network is called to perform image processing on the input image to obtain the image processing result.

7. The method according to claim 6, characterized in that The reference network is a first-round reference network, and determining the i-th-round reference network in the supernetwork includes: Calculate the performance parameters of the i-1th round image processing network and the i-1th round reference network, where i is an integer greater than 1 and not greater than n; The network with better performance parameters between the i-1th round image processing network and the i-1th round reference network is determined as the i-th round reference network.

8. The method according to claim 6, characterized in that The determining of the i-th round reference network in the super network includes: receiving second indication information, where the second indication information is used to indicate the first group of operators in the i-th round constituting the i-th round reference network; The i-th round reference network is determined in the super network according to the second indication information.

9. The method according to any one of claims 1 to 3, characterized in that: The using the training image sample set to train and update the network parameters of the first group of operators and the second group of operators includes: Inputting a first sample in the training image sample set into the random network to obtain a first prediction value; updating the network parameters of the second group of operators based on loss training between the first predicted value and a first true value, where the first true value is a true value corresponding to the first sample; Inputting a second sample in the training image sample set into the reference network to obtain a second prediction value; The network parameters of the first group of operators are updated according to loss training between the second predicted value and the second true value, where the second true value is the true value corresponding to the second sample.

10. The method according to any one of claims 1 to 3, characterized in that: The step of using the verification image sample set to train and update the structural parameters of the at least two candidate operators in the super network includes: Inputting a third sample in the verification image sample set into the hypernetwork to obtain a third prediction value; The structural parameters of the at least two candidate operators in the hypernetwork are updated according to loss training of the third predicted value and a third true value, where the third true value is a true value corresponding to the third sample.

11. The method according to claim 10, characterized in that The super network includes m network layers, the first network layer includes q candidate operators, the jth network layer includes k candidate operators, m, q, k are all positive integers, and j is an integer greater than 1; Inputting the third sample in the verification image sample set into the hypernetwork to obtain a third prediction value includes: Inputting the third sample in the verification image sample set into the q candidate operators in the first network layer respectively to obtain q intermediate outputs; using the structural parameters corresponding to the q candidate operators as weights, calculating the weighted sum of the q intermediate outputs to obtain the first output of the first network layer; The j-1th output of the j-1th network layer is respectively input into the k candidate operators in the jth network layer to obtain k intermediate outputs; the structural parameters corresponding to the k candidate operators are used as weights, and the weighted sum of the k intermediate outputs is calculated to obtain the jth output of the jth network layer; repeat this step to determine the mth output of the mth network layer as the third prediction value.

12. The method according to any one of claims 1 to 3, characterized in that: The method further comprises: Initializing the structural parameters of the supernetwork and the at least two candidate operators; The network parameters of the at least two candidate operators in the super network are trained using a preheated image sample set.

13. An image processing device, characterized in that: The device comprises: a determination module, configured to determine a reference network and a random network in a supernetwork, wherein the supernetwork includes at least two candidate operators, the reference network includes a first set of operators predetermined in the supernetwork, and the random network includes a second set of operators randomly sampled in the supernetwork, wherein the supernetwork is used for image processing; A network training module, configured to train and update network parameters of the first group of operators and the second group of operators using a training image sample set; a structure training module, configured to train and update structure parameters of the at least two candidate operators in the super network using a verification image sample set, wherein the structure parameters are used to determine an image processing network from the super network; A search module, configured to search the image processing network from the super network according to the structural parameters, wherein the image processing network includes a third group of operators in the super network; The calling module is used to call the image processing network to perform image processing on the input image to obtain an image processing result.

14. A computer device, characterized in that: The computer device includes: a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set, or an instruction set, and the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the image processing method according to any one of claims 1 to 12.

15. A computer-readable storage medium, characterized in that The storage medium stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the image processing method according to any one of claims 1 to 12.

Images