Method and apparatus for constructing neural network, and image processing method and apparatus

Abstract

The application discloses a method and device for constructing a neural network and an image processing method and device in the field of artificial intelligence. The method for constructing a neural network comprises the following steps: constructing a search space according to the application requirement of a target neural network, wherein the search space comprises M elements, the M elements are used for indicating M network structures, each element in the M elements comprises the number of blocks in a stage in a corresponding network structure and the channel number of the blocks, and M is a positive integer; and a target network structure is selected from the M network structures according to the distribution relationship of un-evaluated elements in the search space. The method can efficiently construct a neural network meeting performance requirements.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and more specifically, to a method and apparatus for constructing neural networks, and an image processing method and apparatus. Background Technology

[0002] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a branch of computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to have perception, reasoning, and decision-making capabilities. Research in the field of AI includes robotics, natural language processing, computer vision, decision-making and reasoning, human-computer interaction, recommendation and search, and fundamental AI theories.

[0003] With the rapid development of artificial intelligence technology, the performance of neural networks (such as convolutional neural networks) has been continuously improved, and neural networks have also achieved great success in the processing and analysis of various media signals such as images, videos, and audio. A high-performance neural network often has an ingenious network structure, but in practical applications, due to differences in training sets, performance requirements, and application goals, existing network structures often cannot be used directly.

[0004] Currently, a common approach is to adjust the existing network structure for different tasks. However, it is often difficult to obtain a high-performance network structure through adjustment. Another common approach is to automatically search for network structures based on automated machine learning (AutoML). However, AutoML is a method of designing network structures from scratch, and the complexity of the task is very high.

[0005] Therefore, how to efficiently construct neural networks that meet performance requirements has become a pressing technical problem that needs to be solved. Summary of the Invention

[0006] This application provides a method and apparatus for constructing a neural network, as well as an image processing method and apparatus, which can efficiently construct neural networks that meet performance requirements.

[0007] Firstly, a method for constructing a neural network is provided. The method includes: constructing a search space according to the application requirements of the target neural network, the search space including M elements, the M elements being used to indicate M network structures, each of the M elements including the number of blocks in the stage of the corresponding network structure and the number of channels of the block, where M is a positive integer; and selecting a target network structure from the M network structures according to the distribution relationship of unevaluated elements in the search space.

[0008] In this embodiment, representative elements in the search space can be selected based on the distribution relationship of the unevaluated elements. Then, based on these representative elements, the target network structure can be efficiently selected from the M network structures, thereby enabling the efficient construction of a neural network that meets the performance requirements.

[0009] Optionally, each of the M elements including the corresponding network structure refers to the network structure indicated by each of the M elements.

[0010] Accordingly, for each of the M elements, each element includes the number of blocks in the stage of the network structure indicated by the element and the number of channels of the block.

[0011] In conjunction with the first aspect, in some implementations of the first aspect, the application requirements of the target neural network include the running speed of the target neural network, the number of parameters of the target neural network, or the structural requirements of the target neural network, wherein the structural requirements include the number of blocks in each stage of the target neural network structure and the number of channels in each block.

[0012] In this embodiment, a search space is constructed based on the running speed of the target neural network, the number of parameters of the target neural network, or the structural requirements of the target neural network. This allows for the filtering out of some low-performance network structures during the search space construction process, thereby improving the efficiency of neural network construction.

[0013] In conjunction with the first aspect, in some implementations of the first aspect, constructing a search space according to the application requirements of the target neural network includes: constructing an initial search space according to the application requirements of the target neural network, the initial search space including N initial elements, the N initial elements being used to indicate N initial network structures, each of the N initial elements including the number of blocks in a stage of the corresponding initial network structure and the number of channels in the block, where N is a positive integer greater than or equal to M; filtering the N initial network structures indicated by the N initial elements according to a preset rule to obtain the M elements in the search space, the preset rule including: if the number of blocks in each stage of the first initial network structure indicated by the first initial element among the N initial elements is not greater than the number of blocks in the corresponding stage of the second initial network structure indicated by the second initial element among the N initial elements, and the number of channels in each block in each stage of the first initial network structure is not greater than the number of channels in each block in the corresponding stage of the second initial network structure, then the first initial element is deleted from the initial search space.

[0014] In this embodiment of the application, according to the above-mentioned preset rules, without training the initial network structure indicated by the N initial elements in the initial search space, the low-performance network structure among the N initial network structures can be screened out based solely on the structural information of the N initial network structures. Therefore, the efficiency of constructing a neural network can be improved.

[0015] In conjunction with the first aspect, in some implementations of the first aspect, the step of selecting a target network structure from the M network structures based on the distribution relationship of unevaluated elements in the search space includes: determining K elements from the unevaluated elements based on the distribution relationship of the unevaluated elements, where K is a positive integer less than M; and selecting a target network structure from the M network structures based on the K elements.

[0016] In conjunction with the first aspect, in some implementations of the first aspect, the step of selecting a target network structure from the M network structures based on the K elements includes: evaluating the K network structures indicated by the K elements among the unevaluated elements to obtain the evaluation results of the evaluated elements, wherein the evaluation results of the evaluated elements include the evaluation results of the K network structures; and selecting a target network structure from the M network structures based on the evaluation results of the evaluated elements.

[0017] In this embodiment of the application, based on the evaluation results of the evaluated network structures, a target network structure that meets the preset requirements is selected from the M network structures. This allows for full utilization of the evaluation results of the evaluated network structures, thereby improving the efficiency of constructing neural networks.

[0018] In conjunction with the first aspect, in some implementations of the first aspect, the step of selecting a target network structure from the M network structures based on the evaluation results of the evaluated elements includes: modeling a first unevaluated element based on the evaluation results of the evaluated elements to obtain a model of the first unevaluated element, wherein the first unevaluated element includes other elements in the search space besides the evaluated elements; and selecting a target network structure from the M network structures based on the model of the first unevaluated element.

[0019] In this embodiment of the application, the first unevaluated element is modeled based on the evaluation results of the evaluated elements to obtain the model of the first unevaluated element. Using the model of the first unevaluated element helps to select representative elements in the search space. At this time, based on the representative elements, the target network structure can be efficiently selected from the M network structures, thereby enabling the efficient construction of a neural network that meets the performance requirements.

[0020] In conjunction with the first aspect, in some implementations of the first aspect, when a target network structure is not selected from the M network structures based on the distribution relationship of the unevaluated elements in the search space, the method further includes: selecting a target network structure from the M network structures based on the distribution relationship of the first unevaluated elements and a model of the first unevaluated elements constructed based on the evaluated elements.

[0021] In conjunction with the first aspect, in some implementations of the first aspect, the step of selecting a target network structure from the M network structures based on the distribution relationship of the first unevaluated elements and the model of the first unevaluated elements constructed based on the evaluated elements includes: determining L elements from the first unevaluated elements based on the distribution relationship of the first unevaluated elements, where L is a positive integer less than M; and selecting a target network structure from the M network structures based on the L elements and the model of the first unevaluated elements constructed based on the evaluated elements.

[0022] In conjunction with the first aspect, in some implementations of the first aspect, selecting a target network structure from the M network structures based on the L elements and a model of a first unevaluated element constructed based on the evaluated elements includes: determining Q elements from the L elements based on the model of the first unevaluated element constructed based on the evaluated elements, where Q is a positive integer less than L; evaluating the Q network structures indicated by the Q elements to obtain the evaluation result of the first evaluated element, the evaluation result of the first evaluated element including the evaluation results of the K network structures and the evaluation results of the Q network structures; selecting a target network structure from the M network structures based on the distribution relationship between the evaluation result of the first evaluated element and the second unevaluated element, where the second unevaluated element includes other elements in the search space besides the first evaluated element.

[0023] In conjunction with the first aspect, in some implementations of the first aspect, the distribution relationship of the first unevaluated element is the clustering result of the first unevaluated element, and the L elements are elements in the L clusters included in the clustering result of the first unevaluated element.

[0024] In this embodiment, using the clustering results of the first unevaluated element helps to select representative elements in the search space. Then, based on the representative element, the target network structure can be efficiently selected from the M network structures, thereby enabling the efficient construction of a neural network that meets performance requirements.

[0025] In conjunction with the first aspect, in some implementations of the first aspect, the L elements are the L elements corresponding to the centers of the L clusters.

[0026] In this embodiment of the application, the L network structures indicated by the L elements corresponding to the centers of the L clusters are the most representative network structures in the search space. Based on the evaluation results of the Q network structures indicated by the Q elements selected from the L elements, a target network structure that meets the preset requirements is selected from the M network structures. This can efficiently select the target network structure that meets the preset requirements from the M network structures, thereby improving the efficiency of building neural networks.

[0027] In conjunction with the first aspect, in some implementations of the first aspect, when a target network structure is not selected from the M network structures based on the distribution relationship of the first unevaluated elements and the model of the first unevaluated elements constructed based on the evaluated elements, the method further includes: reselecting a target network structure from the M network structures based on the distribution relationship of the second unevaluated elements and the model of the second unevaluated elements constructed based on the first evaluated elements.

[0028] In this embodiment, if a target network structure is not selected from the M network structures based on the distribution relationship of the first unevaluated elements and the model of the first unevaluated elements constructed based on the evaluated elements, a target network structure can be reselected from the M network structures based on the distribution relationship of the second unevaluated elements and the model of the second unevaluated elements constructed based on the first evaluated elements. This can fully utilize the evaluation results of the evaluated network structures, thereby improving the efficiency of building neural networks.

[0029] In conjunction with the first aspect, in some implementations of the first aspect, the model of the first unevaluated element constructed based on the evaluated elements includes: the model of the first unevaluated element is obtained based on the evaluation results of the evaluated network structure and the following formula:

[0030] and / or (g(x)-l(x)) 2 ;

[0031] Where x represents the unevaluated network structure in the search space, y * Let g(x) represent the precision threshold, and g(x) represent precision greater than the precision threshold y. * The conditional probability density function of the network structure x. l(x) represents a precision less than or equal to the precision threshold y. * The conditional probability density function of the network structure x. G * (x|x i ) represents a mixture Gaussian distribution. κ(x, x) i ) represents x and x i The Euclidean distance between them, κ*(x, x i ) represents the expression by κ(x, x i The distance function composed of ) x i Let y represent the i-th network structure in the evaluated network structures. i ω represents the accuracy of the i-th network structure among the evaluated network structures. i G represents * (x|x i The corresponding weights, Z represents the normalization factor, and σ represents G. * (x|x i () is a hyperparameter where i is a positive integer and e is the base of the natural logarithm function.

[0032] In conjunction with the first aspect, in some implementations of the first aspect, the model of the first unevaluated element constructed based on the evaluated elements includes: the model of the first unevaluated element is obtained based on the evaluation results of the evaluated network structure and the following formula:

[0033]

[0034] Where x represents an unevaluated element in the search space; x i Let y represent the i-th element among the evaluated elements. i This represents the precision of the i-th element among the evaluated elements; τ = max(y i ), Let f(x) represent the expectation function; f(x) is a random variable following a Gaussian distribution, and the mean μ(x) and variance σ(x) of f(x) are related to... The input x satisfies the following relationship:

[0035] μ(x)=k T (K+η 2 I) -1 Y,

[0036] σ(x)=1-k T (K+η 2 I) -1 k.

[0037] Where n is the number of network structures evaluated, and Y is y i The vector formed, Y∈R n Y i =y i k is κ(x, x i A vector consisting of k ∈ R n k i =κ(x, x) i ), K is κ(x) i x j The matrix formed by K∈R is given by ( ). n×n K i，j =κ(x) i x j ), σ is a hyperparameter, e is the base of the natural logarithm function; I is the identity matrix, η is also a hyperparameter, and i and j are positive integers.

[0038] In conjunction with the first aspect, in some implementations of the first aspect, the distribution relationship of the unevaluated elements is the clustering result of the unevaluated elements, and the K elements are elements in the K clusters included in the clustering result of the unevaluated elements.

[0039] In this embodiment of the application, using the clustering results of unevaluated elements helps to select representative elements in the search space. At this time, based on the representative elements, the target network structure can be efficiently selected from the M network structures, thereby enabling the efficient construction of a neural network that meets the performance requirements.

[0040] In conjunction with the first aspect, in some implementations of the first aspect, the K elements are the K elements corresponding to the centers of the K clusters.

[0041] In this embodiment of the application, the K network structures indicated by the K elements corresponding to the centers of the K clusters are the most representative network structures in the search space. Based on the evaluation results of the K network structures indicated by the K elements corresponding to the centers of the K clusters, a target network structure that meets the preset requirements is selected from the M network structures. This can efficiently select the target network structure that meets the preset requirements from the M network structures, thereby improving the efficiency of building neural networks.

[0042] Secondly, an image processing method is provided, the method comprising: acquiring an image to be processed; classifying the image to be processed according to a target neural network to obtain a classification result of the image to be processed; wherein the target neural network is constructed from a target network structure, and the target network structure is obtained by any of the methods in the first aspect above.

[0043] It should be understood that the target neural network used in the image processing method in the second aspect needs to be trained again based on the training images before it can perform image classification. The trained target neural network can then classify the images to be processed.

[0044] In other words, the target neural network can be obtained using the neural network structure search method in the first aspect. Then, the target neural network can be trained based on the training images. After training, the target neural network can be used to classify the images to be processed.

[0045] In this application, since the target neural network is constructed using the first aspect mentioned above, it is more in line with or close to the application requirements of neural networks. Using such a neural network for image classification can achieve better image classification results (e.g., more accurate classification results, etc.).

[0046] Thirdly, an apparatus for constructing a neural network is provided, comprising: a construction unit for constructing a search space according to the application requirements of a target neural network, the search space comprising M elements, the M elements being used to indicate M network structures, each of the M elements including the number of blocks in a stage of the corresponding network structure and the number of channels of the block, where M is a positive integer; and a selection unit for selecting a target network structure from the M network structures according to the distribution relationship of unevaluated elements in the search space.

[0047] In this embodiment, representative elements in the search space can be selected based on the distribution relationship of the unevaluated elements. Then, based on these representative elements, the target network structure can be efficiently selected from the M network structures, thereby enabling the efficient construction of a neural network that meets the performance requirements.

[0048] Optionally, each of the M elements including the corresponding network structure refers to the network structure indicated by each of the M elements.

[0049] Accordingly, for each of the M elements, each element includes the number of blocks in the stage of the network structure indicated by the element and the number of channels of the block.

[0050] In conjunction with the third aspect, in some implementations of the third aspect, the application requirements of the target neural network include the running speed of the target neural network, the number of parameters of the target neural network, or the structural requirements of the target neural network, wherein the structural requirements include the number of blocks in each stage of the target neural network structure and the number of channels in each block.

[0051] In this embodiment, a search space is constructed based on the running speed of the target neural network, the number of parameters of the target neural network, or the structural requirements of the target neural network. This allows for the filtering out of some low-performance network structures during the search space construction process, thereby improving the efficiency of neural network construction.

[0052] In conjunction with the third aspect, in some implementations of the third aspect, the construction unit is specifically used to: construct an initial search space according to the application requirements of the target neural network, the initial search space including N initial elements, the N initial elements being used to indicate N initial network structures, each of the N initial elements including the number of blocks in a stage of the corresponding initial network structure and the number of channels of the block, where N is a positive integer greater than or equal to M; filter the N initial network structures indicated by the N initial elements according to preset rules to obtain the M elements in the search space, the preset rules including: if the number of blocks in each stage of the first initial network structure indicated by the first initial element among the N initial elements is not greater than the number of blocks in the corresponding stage of the second initial network structure indicated by the second initial element among the N initial elements, and the number of channels of each block in each stage of the first initial network structure is not greater than the number of channels of each block in the corresponding stage of the second initial network structure, then delete the first initial element from the initial search space.

[0053] In this embodiment of the application, according to the above-mentioned preset rules, without training the initial network structure indicated by the N initial elements in the initial search space, the low-performance network structure among the N initial network structures can be screened out based solely on the structural information of the N initial network structures. Therefore, the efficiency of constructing a neural network can be improved.

[0054] In conjunction with the third aspect, in some implementations of the third aspect, the selection unit is specifically used to: determine K elements among the unevaluated elements based on the distribution relationship of the unevaluated elements, where K is a positive integer less than M; and select a target network structure from the M network structures based on the K elements.

[0055] In conjunction with the third aspect, in some implementations of the third aspect, the selection unit is specifically used to: evaluate the K network structures indicated by the K elements among the unevaluated elements to obtain the evaluation results of the evaluated elements, wherein the evaluation results of the evaluated elements include the evaluation results of the K network structures; and select a target network structure from the M network structures based on the evaluation results of the evaluated elements.

[0056] In this embodiment of the application, based on the evaluation results of the evaluated network structures, a target network structure that meets the preset requirements is selected from the M network structures. This allows for full utilization of the evaluation results of the evaluated network structures, thereby improving the efficiency of constructing neural networks.

[0057] In conjunction with the third aspect, in some implementations of the third aspect, the selection unit is specifically used to: model the first unevaluated element based on the evaluation result of the evaluated element to obtain a model of the first unevaluated element, wherein the first unevaluated element includes other elements in the search space besides the evaluated element; and select a target network structure from the M network structures based on the model of the first unevaluated element.

[0058] In this embodiment of the application, the first unevaluated element is modeled based on the evaluation results of the evaluated elements to obtain the model of the first unevaluated element. Using the model of the first unevaluated element helps to select representative elements in the search space. At this time, based on the representative elements, the target network structure can be efficiently selected from the M network structures, thereby enabling the efficient construction of a neural network that meets the performance requirements.

[0059] In conjunction with the third aspect, in some implementations of the third aspect, when a target network structure is not selected from the M network structures based on the distribution relationship of the unevaluated elements in the search space, the selection unit is further configured to: select a target network structure from the M network structures based on the distribution relationship of the first unevaluated elements and the model of the first unevaluated elements constructed based on the evaluated elements.

[0060] In conjunction with the third aspect, in some implementations of the third aspect, the selection unit is specifically used to: determine L elements from the first unevaluated elements according to the distribution relationship of the first unevaluated elements, where L is a positive integer less than M; and select a target network structure from the M network structures based on the L elements and the model of the first unevaluated elements constructed based on the evaluated elements.

[0061] In conjunction with the third aspect, in some implementations of the third aspect, the selection unit is specifically used for: determining Q elements from the L elements based on the model of the first unevaluated elements constructed based on the evaluated elements, where Q is a positive integer less than L; evaluating the Q network structures indicated by the Q elements to obtain the evaluation result of the first evaluated element, the evaluation result of the first evaluated element including the evaluation results of the K network structures and the evaluation results of the Q network structures; and selecting a target network structure from the M network structures based on the distribution relationship between the evaluation result of the first evaluated element and the second unevaluated element, where the second unevaluated element includes other elements in the search space besides the first evaluated element.

[0062] In conjunction with the third aspect, in some implementations of the third aspect, the distribution relationship of the first unevaluated element is the clustering result of the first unevaluated element, and the L elements are elements in the L clusters included in the clustering result of the first unevaluated element.

[0063] In this embodiment, using the clustering results of the first unevaluated element helps to select representative elements in the search space. Then, based on the representative element, the target network structure can be efficiently selected from the M network structures, thereby enabling the efficient construction of a neural network that meets performance requirements.

[0064] In conjunction with the third aspect, in some implementations of the third aspect, the L elements are the L elements corresponding to the centers of the L clusters.

[0065] In this embodiment of the application, the L network structures indicated by the L elements corresponding to the centers of the L clusters are the most representative network structures in the search space. Based on the evaluation results of the Q network structures indicated by the Q elements determined from the L elements, a target network structure that meets the preset requirements is selected from the M network structures. This can efficiently select the target network structure that meets the preset requirements from the M network structures, thereby improving the efficiency of building neural networks.

[0066] In conjunction with the third aspect, in some implementations of the third aspect, when a target network structure is not selected from the M network structures based on the distribution relationship of the first unevaluated elements and the model of the first unevaluated elements constructed based on the evaluated elements, the selection unit is further configured to: reselect a target network structure from the M network structures based on the distribution relationship of the second unevaluated elements and the model of the second unevaluated elements constructed based on the first evaluated elements.

[0067] In this embodiment, if a target network structure is not selected from the M network structures based on the distribution relationship of the first unevaluated elements and the model of the first unevaluated elements constructed based on the evaluated elements, a target network structure can be reselected from the M network structures based on the distribution relationship of the second unevaluated elements and the model of the second unevaluated elements constructed based on the first evaluated elements. This can fully utilize the evaluation results of the evaluated network structures, thereby improving the efficiency of building neural networks.

[0068] In conjunction with the third aspect, in some implementations of the third aspect, the selection unit is specifically used for: the model of the first unevaluated element is obtained based on the evaluation results of the evaluated network structure and the following formula:

[0069] and / or (g(x)-l(x)) 2 ;

[0070] Where x represents the unevaluated network structure in the search space, y * Let g(x) represent the precision threshold, and g(x) represent precision greater than the precision threshold y. * The conditional probability density function of the network structure x. l(x) represents a precision less than or equal to the precision threshold y. * The conditional probability density function of the network structure x. G * (x|x i ) represents a mixture Gaussian distribution. κ(x, x) i ) represents x and x i The Euclidean distance between them, κ*(x, x i ) represents the expression by κ(x, x i The distance function composed of ) x i Let y represent the i-th network structure in the evaluated network structures. i ω represents the accuracy of the i-th network structure among the evaluated network structures. i G represents * (x|x i The corresponding weights, Z represents the normalization factor, and σ represents G. * (x|x i () is a hyperparameter where i is a positive integer and e is the base of the natural logarithm function.

[0071] In conjunction with the third aspect, in some implementations of the third aspect, the selection unit is specifically used for: the model of the first unevaluated element is obtained based on the evaluation results of the evaluated network structure and the following formula:

[0072]

[0073] Where x represents an unevaluated element in the search space; x i Let y represent the i-th element among the evaluated elements. i This represents the precision of the i-th element among the evaluated elements; τ = max(y i ), Let f(x) represent the expectation function; f(x) is a random variable following a Gaussian distribution, and the mean μ(x) and variance σ(x) of f(x) are related to... The input x satisfies the following relationship:

[0074] μ(x)=k T (K+η 2I) -1 Y,

[0075] σ(x)=1-k T (K+η 2 I) -1 k.

[0076] Where n is the number of network structures evaluated, and Y is y i The vector formed, Y∈R n Y i =y i k is κ(x, x i A vector consisting of k ∈ R n k i =κ(x, x) i ), K is κ(x) i x j The matrix formed by K∈R is given by ( ). n×n K i，j =κ(x) i x j ), σ is a hyperparameter, e is the base of the natural logarithm function; I is the identity matrix, η is also a hyperparameter, and i and j are positive integers.

[0077] In conjunction with the third aspect, in some implementations of the third aspect, the distribution relationship of the unevaluated elements is the clustering result of the unevaluated elements, and the K elements are elements in the K clusters included in the clustering result of the unevaluated elements.

[0078] In this embodiment of the application, using the clustering results of unevaluated elements helps to select representative elements in the search space. At this time, based on the representative elements, the target network structure can be efficiently selected from the M network structures, thereby enabling the efficient construction of a neural network that meets the performance requirements.

[0079] In conjunction with the third aspect, in some implementations of the third aspect, the K elements are the K elements corresponding to the centers of the K clusters.

[0080] In this embodiment of the application, the K network structures indicated by the K elements corresponding to the centers of the K clusters are the most representative network structures in the search space. Based on the evaluation results of the K network structures indicated by the K elements corresponding to the centers of the K clusters, a target network structure that meets the preset requirements is selected from the M network structures. This can efficiently select the target network structure that meets the preset requirements from the M network structures, thereby improving the efficiency of building neural networks.

[0081] Fourthly, an image processing apparatus is provided, comprising: an acquisition unit for acquiring an image to be processed; and an image processing unit for classifying the image to be processed according to a target neural network to obtain a classification result of the image to be processed; wherein the target neural network is constructed from a target network structure, and the target network structure is obtained by any of the methods in the first aspect described above.

[0082] It should be understood that the target neural network used in the image processing method in the second aspect needs to be trained again based on the training images before it can perform image classification. The trained target neural network can then classify the images to be processed.

[0083] In other words, the target neural network can be obtained using the neural network structure search method in the first aspect. Then, the target neural network can be trained based on the training images. After training, the target neural network can be used to classify the images to be processed.

[0084] In this application, since the target neural network is constructed using the first aspect mentioned above, it is more in line with or close to the application requirements of neural networks. Using such a neural network for image classification can achieve better image classification results (e.g., more accurate classification results, etc.).

[0085] It should be understood that the extensions, limitations, interpretations and descriptions of the relevant content in the first aspect above also apply to the same content in the second, third and fourth aspects.

[0086] Fifthly, an apparatus for constructing a neural network is provided, the apparatus comprising: a memory for storing a program; and a processor for executing the program stored in the memory, wherein when the program stored in the memory is executed, the processor is configured to execute a method in any implementation of the first aspect.

[0087] The processor mentioned in the fifth aspect above can be a central processing unit (CPU) or a combination of a CPU and a neural network processing processor. This neural network processing processor can include a graphics processing unit (GPU), a neural network processing unit (NPU), and a tensor processing unit (TPU), etc. The TPU is a dedicated integrated circuit from Google, designed entirely for machine learning and artificial intelligence accelerators.

[0088] In a sixth aspect, an image processing apparatus is provided, comprising: a memory for storing a program; and a processor for executing the program stored in the memory, wherein when the program stored in the memory is executed, the processor is configured to execute a method in any implementation of the second aspect.

[0089] The processor mentioned in the sixth aspect above can be a central processing unit (CPU) or a combination of a CPU and a neural network processing processor. This neural network processing processor can include a graphics processing unit (GPU), a neural network processing unit (NPU), and a tensor processing unit (TPU), etc. The TPU is a dedicated integrated circuit from Google, designed entirely for machine learning and artificial intelligence accelerators.

[0090] A seventh aspect provides a computer-readable medium storing program code for execution by a device, the program code including methods for performing any implementation of the first or second aspect.

[0091] Eighthly, a computer program product containing instructions is provided, which, when run on a computer, causes the computer to perform the method in any one of the implementations of the first or second aspect described above.

[0092] Ninth aspect, a chip is provided, the chip including a processor and a data interface, the processor reading instructions stored in a memory through the data interface and executing the method in any one of the implementations of the first or second aspect described above.

[0093] Optionally, as one implementation, the chip may further include a memory storing instructions, and the processor is used to execute the instructions stored in the memory. When the instructions are executed, the processor is used to execute the method in either the first aspect or the second aspect.

[0094] The aforementioned chip can be a field-programmable gate array (FPGA) or an application-specific integrated circuit (ASIC).

[0095] In this embodiment, representative elements in the search space can be selected based on the distribution relationship of the unevaluated elements. Then, based on these representative elements, the target network structure can be efficiently selected from the M network structures, thereby enabling the efficient construction of a neural network that meets the performance requirements. Attached Figure Description

[0096] Figure 1 This is a schematic diagram of an artificial intelligence main framework provided in an embodiment of this application.

[0097] Figure 2 This is a schematic diagram of a system architecture provided in an embodiment of this application.

[0098] Figure 3 This is a schematic diagram of the structure of a convolutional neural network provided in an embodiment of this application.

[0099] Figure 4 This is a schematic diagram of another convolutional neural network structure provided in an embodiment of this application.

[0100] Figure 5 This is a schematic diagram of the hardware structure of a chip provided in an embodiment of this application.

[0101] Figure 6 This is a schematic diagram of a system architecture provided in an embodiment of this application.

[0102] Figure 7 This is a schematic flowchart of a method for constructing a neural network according to an embodiment of this application.

[0103] Figure 8 This is a schematic block diagram of a residual block provided in one embodiment of this application.

[0104] Figure 9 This is a schematic flowchart of a method for constructing a neural network according to an embodiment of this application.

[0105] Figure 10 This is a schematic block diagram of the network structure in the search space provided in one embodiment of this application.

[0106] Figure 11 This is a schematic flowchart of an image processing method according to an embodiment of this application.

[0107] Figure 12 This is a schematic block diagram of an apparatus for constructing a neural network according to an embodiment of this application.

[0108] Figure 13 This is a schematic block diagram of an image processing apparatus according to an embodiment of this application.

[0109] Figure 14 This is a schematic block diagram of a neural network training apparatus according to an embodiment of this application. Detailed Implementation

[0110] The technical solutions of the embodiments of this application will be described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0111] Figure 1 A schematic diagram of an artificial intelligence framework is shown, which describes the overall workflow of an artificial intelligence system and is applicable to general artificial intelligence domain needs.

[0112] The above-mentioned artificial intelligence framework will be elaborated in detail from two dimensions: "intelligent information chain" (horizontal axis) and "information technology (IT) value chain" (vertical axis).

[0113] The "intelligent information chain" reflects a series of processes from data acquisition to processing. For example, it could be a general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, and intelligent execution and output. In this process, data undergoes a condensation process of "data-information-knowledge-wisdom."

[0114] The "IT value chain" reflects the value that artificial intelligence brings to the information technology industry, from the underlying infrastructure of human intelligence, information (provided and processed by technology) to the industrial ecosystem of systems.

[0115] (1) Infrastructure:

[0116] Infrastructure provides computing power to support artificial intelligence systems, enables them to communicate with the outside world, and provides support through basic platforms.

[0117] Infrastructure can communicate with the outside world through sensors, and its computing power can be provided by smart chips.

[0118] The intelligent chips here can be hardware acceleration chips such as central processing units (CPUs), neural network processing units (NPUs), graphics processing units (GPUs), application-specific integrated circuits (ASICs), and field-programmable gate arrays (FPGAs).

[0119] The basic platform of the infrastructure can include distributed computing frameworks and related platform guarantees and support, such as cloud storage and computing, and interconnected networks.

[0120] For example, for infrastructure, data can be acquired through sensors and external communication, and then this data can be provided to intelligent chips in the distributed computing system provided by the basic platform for computation.

[0121] (2) Data:

[0122] The data at the next layer of infrastructure is used to represent data sources in the field of artificial intelligence. This data includes graphics, images, voice, text, and IoT data from traditional devices, including business data from existing systems and sensor data such as force, displacement, liquid level, temperature, and humidity.

[0123] (3) Data processing:

[0124] The aforementioned data processing typically includes data training, machine learning, deep learning, search, reasoning, and decision-making.

[0125] Among them, machine learning and deep learning can perform intelligent information modeling, extraction, preprocessing, and training on data, including symbolization and formalization.

[0126] Reasoning refers to the process in which, in a computer or intelligent system, the machine thinks and solves problems by simulating human intelligent reasoning, based on reasoning control strategies and using formalized information. Typical functions include search and matching.

[0127] Decision-making refers to the process of making decisions based on intelligent information after reasoning, and it typically provides functions such as classification, sorting, and prediction.

[0128] (4) General abilities:

[0129] After the data processing mentioned above, the results of the data processing can be used to form some general capabilities, such as algorithms or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc.

[0130] (5) Smart products and industry applications:

[0131] Intelligent products and industry applications refer to products and applications of artificial intelligence systems in various fields. They encapsulate overall artificial intelligence solutions, productize intelligent information decision-making, and realize practical applications. Their application areas mainly include: intelligent manufacturing, intelligent transportation, smart home, intelligent healthcare, intelligent security, autonomous driving, safe city, and intelligent terminals.

[0132] The embodiments of this application can be applied to many fields of artificial intelligence, such as intelligent manufacturing, intelligent transportation, smart home, intelligent healthcare, intelligent security, autonomous driving, and safe city.

[0133] Specifically, the embodiments of this application can be applied to fields that require the use of (deep) neural networks, such as autonomous driving, image classification, image retrieval, image semantic segmentation, image quality enhancement, image super-resolution, and natural language processing.

[0134] The following is a brief introduction to two application scenarios: photo album categorization and autonomous driving.

[0135] Photo album categories:

[0136] Specifically, when users store a large number of pictures on terminal devices (e.g., mobile phones) or cloud storage, recognizing the images in the album can facilitate users or the system to classify and manage the album, thereby improving the user experience.

[0137] Using the neural network construction method of this application embodiment, a neural network suitable for album classification is constructed. Then, the neural network is trained using training images from a training image library to obtain the album classification neural network. This album classification neural network can then be used to classify images, thereby tagging images of different categories for easy viewing and searching by users. Furthermore, these image classification tags can be provided to the album management system for categorization management, saving users' management time, improving album management efficiency, and enhancing user experience.

[0138] Object recognition in autonomous driving scenarios:

[0139] Autonomous driving involves processing a large amount of sensor data, and deep neural networks play a crucial role due to their powerful capabilities. However, manually designing the corresponding data processing network is time-consuming and labor-intensive. Therefore, by adopting the neural network construction method of the embodiments of this application, a neural network suitable for data processing in autonomous driving scenarios can be constructed. Next, the neural network is trained using data from autonomous driving scenarios to obtain a sensor data processing network. Finally, this sensor processing network can be used to process the input road images, thereby identifying different objects in the road images.

[0140] Since the embodiments of this application involve a large number of neural network applications, for ease of understanding, the relevant terms and concepts of neural networks that may be involved in the embodiments of this application will be introduced below.

[0141] (1) Neural Network

[0142] Neural networks can be composed of neural units, which can refer to units represented by x. s The arithmetic unit that takes an intercept of 1 as input can output the following:

[0143]

[0144] Where s = 1, 2, ..., n, n is a natural number greater than 1, W s For x s The weights are denoted by b, where b is the bias of the neural unit. f is the activation function of the neural unit, used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into an output signal. The output signal of this activation function can be used as the input of the next convolutional layer; the activation function can be the sigmoid function. A neural network is a network formed by connecting multiple of the above-mentioned individual neural units together; that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field, which can be a region composed of several neural units.

[0145] (2) Deep Neural Networks

[0146] A deep neural network (DNN), also known as a multilayer neural network, can be understood as a neural network with multiple hidden layers. Based on the position of the layers, the internal neural network of a DNN can be divided into three categories: input layer, hidden layer, and output layer. Generally, the first layer is the input layer, the last layer is the output layer, and the layers in between are hidden layers. The layers are fully connected, meaning that any neuron in the i-th layer is connected to any neuron in the (i+1)-th layer.

[0147] Although DNNs seem complex, the operation of each layer is actually not complicated. Simply put, it involves the following linear relationship expression: in, It is the input vector. It is the output vector. α is the offset vector, W is the weight matrix (also called coefficients), and α() is the activation function. Each layer is simply an adjustment of the input vector. The output vector is obtained through such a simple operation. Because DNNs have many layers, the coefficients W and the offset vector... The number of these parameters is also relatively large. The definitions of these parameters in DNNs are as follows: Taking the coefficient W as an example: Assuming a three-layer DNN, the linear coefficient from the 4th neuron in the second layer to the 2nd neuron in the third layer is defined as... The superscript 3 represents the layer number where coefficient W is located, while the subscript corresponds to the third layer index 2 of the output and the second layer index 4 of the input.

[0148] In summary, the coefficient from the k-th neuron in layer L-1 to the i-th neuron in layer L is defined as...

[0149] It's important to note that the input layer does not have a W parameter. In deep neural networks, more hidden layers allow the network to better represent complex real-world situations. Theoretically, the more parameters a model has, the higher its complexity and "capacity," meaning it can perform more complex learning tasks. Training a deep neural network is essentially the process of learning the weight matrix, with the ultimate goal of obtaining the weight matrix of all layers in the trained deep neural network (a weight matrix formed by the vectors W from many layers).

[0150] (3) Convolutional Neural Network

[0151] A convolutional neural network (CNN) is a deep neural network with a convolutional structure. A CNN contains a feature extractor consisting of convolutional layers and subsampling layers, which can be viewed as a filter. A convolutional layer is a layer of neurons in a CNN that performs convolutional processing on the input signal. In a convolutional layer of a CNN, a neuron may only be connected to some of its neighboring neurons. A convolutional layer typically contains several feature planes, each composed of a series of rectangularly arranged neural units. Neural units on the same feature plane share weights, which are called the convolutional kernel. Shared weights can be understood as the way image information is extracted regardless of location. The convolutional kernel can be initialized as a matrix of random size, and during the training process of the CNN, the kernel can learn appropriate weights. Furthermore, the direct benefit of shared weights is that it reduces the connections between layers in the CNN, while also reducing the risk of overfitting.

[0152] (4) Recurrent neural networks (RNNs) are used to process sequential data. In traditional neural network models, the layers are fully connected from the input layer to the hidden layer and then to the output layer, but the nodes within each layer are unconnected. While this type of ordinary neural network has solved many difficult problems, it is still powerless against many others. For example, to predict the next word in a sentence, you generally need to use the preceding words because the words in a sentence are not independent. RNNs are called recurrent neural networks because the current output of a sequence is related to the previous output. Specifically, the network memorizes previous information and applies it to the calculation of the current output. That is, the nodes within the hidden layer are no longer unconnected but connected, and the input of the hidden layer includes not only the output of the input layer but also the output of the hidden layer at the previous time step. Theoretically, RNNs can process sequential data of any length. Training an RNN is the same as training a traditional CNN or DNN.

[0153] Since we already have convolutional neural networks (CNNs), why do we need recurrent neural networks (RNNs)? The reason is simple. CNNs rely on the fundamental assumption that elements are independent of each other, and that input and output are also independent—like a cat and a dog. However, in the real world, many elements are interconnected. For example, stock prices fluctuate over time. Or, imagine someone saying, "I love traveling, and my favorite place is Yunnan. I definitely want to go there someday." Humans know the answer to this question is "Yunnan." Humans can infer from context, but how can machines do the same? This is where RNNs come in. RNNs aim to give machines the ability to remember, just like humans. Therefore, the output of an RNN depends on both the current input information and historical memory information.

[0154] (5) Loss Function

[0155] In training a deep neural network, to ensure the output closely approximates the desired predicted value, we compare the network's prediction with the target value. Based on the difference, we update the weight vector of each layer (usually pre-configuring parameters before the initial update). For example, if the prediction is too high, the weight vector is adjusted to predict a lower value. This adjustment continues until the deep neural network predicts the target value or a value very close to it. Therefore, we need to predefine "how to compare the difference between the predicted and target values," which is the loss function or objective function. These are important equations used to measure the difference between the predicted and target values. Taking the loss function as an example, a higher output value (loss) indicates a greater difference, and training the deep neural network becomes a process of minimizing this loss.

[0156] (6) Backpropagation algorithm

[0157] Neural networks can employ backpropagation (BP) to correct the parameters of the initial neural network model during training, thereby reducing the reconstruction error loss. Specifically, forward propagation of the input signal to the output generates error loss; this error loss information is then propagated back to update the parameters of the initial neural network model, leading to convergence of the error loss. The backpropagation algorithm is an error-loss-driven backpropagation process aimed at obtaining the optimal parameters of the neural network model, such as the weight matrix.

[0158] like Figure 2 As shown, this application embodiment provides a system architecture 100. In Figure 2 In this embodiment, the data acquisition device 160 is used to acquire training data. For the image processing method of this application, the training data may include training images and corresponding classification results, wherein the results of the training images may be manually pre-annotated results.

[0159] After collecting the training data, the data acquisition device 160 stores the training data in the database 130, and the training device 120 trains the target model / rule 101 based on the training data maintained in the database 130.

[0160] The following describes how the training device 120 obtains the target model / rule 101 based on the training data. The training device 120 processes the input original image and compares the output image with the original image until the difference between the output image of the training device 120 and the original image is less than a certain threshold, thereby completing the training of the target model / rule 101.

[0161] The aforementioned target model / rule 101 can be used to implement the image processing method of this application embodiment. Specifically, the target model / rule 101 in this application embodiment can be a neural network. It should be noted that in practical applications, the training data maintained in the database 130 may not all come from the data acquisition device 160; it may also be received from other devices. Furthermore, it should be noted that the training device 120 may not necessarily train the target model / rule 101 entirely based on the training data maintained in the database 130; it may also obtain training data from the cloud or other sources for model training. The above description should not be construed as limiting the embodiments of this application.

[0162] The target model / rule 101 trained using training device 120 can be applied to different systems or devices, such as... Figure 2 The execution device 110 shown can be a terminal, such as a mobile phone terminal, tablet computer, laptop computer, augmented reality (AR) / virtual reality (VR) device, vehicle terminal, etc., or it can be a server or cloud service. Figure 2 In this embodiment, the execution device 110 is configured with an input / output (I / O) interface 112 for data interaction with external devices. Users can input data to the I / O interface 112 through the client device 140. The input data may include the image to be processed input by the client device.

[0163] Preprocessing module 113 and preprocessing module 114 are used to preprocess the input data (such as the image to be processed) received by I / O interface 112. In this embodiment, preprocessing module 113 and preprocessing module 114 may be omitted (or only one of them may be used), and the calculation module 111 may be used directly to process the input data.

[0164] During the preprocessing of input data by the execution device 110, or during the calculation module 111 of the execution device 110 performing calculations and other related processes, the execution device 110 can call data, code, etc. in the data storage system 150 for corresponding processing, or store the data, instructions, etc. obtained from the corresponding processing into the data storage system 150.

[0165] Finally, I / O interface 112 returns the processing results, such as the image classification results obtained above, to client device 140, thereby providing them to the user.

[0166] It is worth noting that the training device 120 can generate corresponding target models / rules 101 based on different training data for different objectives or tasks. The corresponding target models / rules 101 can be used to achieve the above objectives or complete the above tasks, thereby providing the user with the required results.

[0167] exist Figure 2 In the scenario shown, the user can manually provide input data, which can be done through the interface provided by I / O interface 112. Alternatively, the client device 140 can automatically send input data to I / O interface 112. If user authorization is required for the client device 140 to automatically send input data, the user can set the corresponding permissions in the client device 140. The user can view the output results of the execution device 110 on the client device 140, which can be presented in various forms such as display, sound, or animation. The client device 140 can also act as a data acquisition terminal, collecting the input data and output results of the input I / O interface 112 as new sample data and storing them in the database 130. Alternatively, data can be collected directly from the I / O interface 112 without going through the client device 140, using the input data and output results of the input I / O interface 112 as new sample data and storing them in the database 130.

[0168] It is worth noting that, Figure 2 This is merely a schematic diagram of a system architecture provided in an embodiment of this application. The positional relationships between the devices, components, modules, etc., shown in the diagram do not constitute any limitation. For example, in Figure 2In this context, the data storage system 150 is an external memory relative to the execution device 110. In other cases, the data storage system 150 may also be placed within the execution device 110.

[0169] like Figure 2 As shown, the target model / rule 101 is obtained by training according to the training device 120. The target model / rule 101 can be the neural network in this application embodiment. Specifically, the neural network constructed in this application embodiment can be CNN, deep convolutional neural network (DCNN), recurrent neural network (RNNS), etc.

[0170] Since CNN is a very common type of neural network, the following will combine... Figure 3 This section focuses on a detailed explanation of the structure of CNNs. As mentioned in the basic concept introduction above, a Convolutional Neural Network (CNN) is a deep neural network with a convolutional structure. It is a deep learning architecture, which refers to learning at multiple levels of abstraction through machine learning algorithms. As a deep learning architecture, CNN is a feed-forward artificial neural network, where each neuron responds to the input image.

[0171] The image processing method in this application embodiment specifically employs a neural network structure as follows: Figure 3 As shown. In Figure 3 In this example, the Convolutional Neural Network (CNN) 200 may include an input layer 210, convolutional / pooling layers 220 (where the pooling layers are optional), and a neural network layer 230. The input layer 210 acquires the image to be processed and then passes the acquired image to the convolutional / pooling layers 220 and the subsequent neural network layer 230 for processing to obtain the image processing result. The following section... Figure 3 This section provides a detailed introduction to the internal layer structure of CNN 200.

[0172] Convolutional / pooling layers 220:

[0173] Convolutional layers:

[0174] like Figure 3The convolutional / pooling layer 220 shown may include layers as in Examples 221-226. For instance, in one implementation, layer 221 is a convolutional layer, layer 222 is a pooling layer, layer 223 is a convolutional layer, layer 224 is a pooling layer, layer 225 is a convolutional layer, and layer 226 is a pooling layer; in another implementation, layers 221 and 222 are convolutional layers, layer 223 is a pooling layer, layers 224 and 225 are convolutional layers, and layer 226 is a pooling layer. That is, the output of the convolutional layer can be used as the input to a subsequent pooling layer, or as the input to another convolutional layer to continue the convolution operation.

[0175] The following section will use convolutional layer 221 as an example to introduce the internal working principle of a convolutional layer.

[0176] Convolutional layer 221 can include multiple convolution operators, also known as kernels. In image processing, a convolution operator acts as a filter to extract specific information from the input image matrix. Essentially, a convolution operator can be a weight matrix, which is usually predefined. During the convolution operation, the weight matrix typically processes the input image pixel by pixel (or two pixels by two pixels, depending on the stride) along the horizontal direction, thus extracting specific features from the image. The size of the weight matrix should be related to the image size. It's important to note that the depth dimension of the weight matrix is ​​the same as the depth dimension of the input image; during convolution, the weight matrix extends to the entire depth of the input image. Therefore, convolution with a single weight matrix produces a single-depth convolutional output. However, in most cases, a single weight matrix is ​​not used; instead, multiple weight matrices of the same size (rows × columns) are applied—multiple identical matrices. The outputs of each weight matrix are stacked to form the depth dimension of the convolutional image; this dimension can be understood as being determined by the "multiple" mentioned above. Different weight matrices can be used to extract different features from an image. For example, one weight matrix can be used to extract image edge information, another weight matrix can be used to extract specific colors of the image, and yet another weight matrix can be used to blur unwanted noise in the image. These multiple weight matrices have the same size (rows × columns), and the resulting convolutional feature maps extracted by these weight matrices also have the same size. These multiple convolutional feature maps with the same size are then merged to form the output of the convolution operation.

[0177] The weight values ​​in these weight matrices need to be obtained through extensive training in practical applications. The weight matrices formed by the weight values ​​obtained through training can be used to extract information from the input image, thereby enabling the convolutional neural network 200 to make correct predictions.

[0178] When a convolutional neural network 200 has multiple convolutional layers, the initial convolutional layers (e.g., 221) tend to extract more general features, which can also be called low-level features. As the depth of the convolutional neural network 200 increases, the features extracted by later convolutional layers (e.g., 226) become more and more complex, such as high-level semantic features. Features with higher semantic levels are more suitable for the problem to be solved.

[0179] Pooling layer:

[0180] Because it is often necessary to reduce the number of training parameters, pooling layers are often introduced periodically after convolutional layers, such as... Figure 3 Layers 221-226 in example 220 can be a convolutional layer followed by a pooling layer, or multiple convolutional layers followed by one or more pooling layers. In image processing, the sole purpose of pooling layers is to reduce the spatial size of the image. Pooling layers can include average pooling and / or max pooling operators to sample the input image to obtain a smaller image size. The average pooling operator calculates the average value of pixel values ​​within a specific range as the result of average pooling. The max pooling operator takes the pixel with the largest value within a specific range as the result of max pooling. Furthermore, just as the size of the weight matrix in a convolutional layer should be related to the image size, the operators in a pooling layer should also be related to the image size. The size of the output image after processing by the pooling layer can be smaller than the size of the input image of the pooling layer. Each pixel in the output image of the pooling layer represents the average or maximum value of the corresponding sub-region of the input image of the pooling layer.

[0181] Neural network layer 230:

[0182] After processing by the convolutional / pooling layers 220, the convolutional neural network 200 is still insufficient to output the required information. As mentioned earlier, the convolutional / pooling layers 220 only extract features and reduce the parameters introduced by the input image. However, to generate the final output information (the required class information or other relevant information), the convolutional neural network 200 needs to utilize neural network layers 230 to generate one or more outputs representing the required number of classes. Therefore, neural network layers 230 can include multiple hidden layers (such as...). Figure 3 As shown in layers 231, 232 to 23n) and output layer 240, the parameters contained in these multi-layer hidden layers can be pre-trained based on relevant training data for specific task types, such as image recognition, image classification, image super-resolution reconstruction, etc.

[0183] After the multiple hidden layers in neural network layer 230, the final layer of the entire convolutional neural network 200 is the output layer 240. This output layer 240 has a loss function similar to classification cross-entropy, specifically used to calculate the prediction error. Once the entire convolutional neural network 200 has undergone forward propagation (e.g., ...), the loss function is applied. Figure 3 Propagation from 210 to 240 degrees is considered forward propagation, while backward propagation (e.g.) is completed. Figure 3 The propagation from 240 to 210 (backpropagation) will begin to update the weight values ​​and biases of the layers mentioned above, in order to reduce the loss of the convolutional neural network 200 and the error between the output of the convolutional neural network 200 through the output layer and the ideal result.

[0184] The image processing method in this application embodiment specifically employs a neural network structure as follows: Figure 4 As shown. In Figure 4 In this example, the Convolutional Neural Network (CNN) 200 may include an input layer 110, convolutional / pooling layers 120 (where pooling layers are optional), and neural network layers 130. Figure 3 compared to, Figure 4 Multiple convolutional / pooling layers in the convolutional / pooling layer 120 are processed in parallel, and the extracted features are input into the full neural network layer 130 for processing.

[0185] It should be noted that, Figure 3 and Figure 4 The convolutional neural networks shown are merely examples of two possible convolutional neural networks for the image processing method of this application embodiment. In specific applications, the convolutional neural networks used in the image processing method of this application embodiment may also exist in the form of other network models.

[0186] Furthermore, the structure of the convolutional neural network obtained using the neural network structure search method of the embodiments of this application can be as follows: Figure 3 and Figure 4 The convolutional neural network structure is shown in the figure.

[0187] Figure 5 The present application provides a hardware structure for a chip, which includes a neural network processor 50. This chip can be configured as follows: Figure 1 The execution device 110 shown is used to perform the calculations of the calculation module 111. This chip can also be located in, for example... Figure 1 The training device 120 shown is used to complete the training work of the training device 120 and output the target model / rule 101. For example... Figure 3 and Figure 4 The algorithms for each layer in the convolutional neural network shown can all be implemented in, for example... Figure 5 This is achieved in the chip shown.

[0188] The neural network processor NPU50 is mounted as a coprocessor on the main central processing unit (CPU) (host CPU), and tasks are assigned by the host CPU. The core of the NPU is the arithmetic circuit 503, and the controller 504 controls the arithmetic circuit 503 to retrieve data from the memory (weight memory or input memory) and perform calculations.

[0189] In some implementations, the arithmetic circuit 503 internally includes multiple process engines (PEs). In some implementations, the arithmetic circuit 503 is a two-dimensional pulsating array. The arithmetic circuit 503 can also be a one-dimensional pulsating array or other electronic circuitry capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit 503 is a general-purpose matrix processor.

[0190] For example, suppose we have an input matrix A, a weight matrix B, and an output matrix C. The arithmetic circuit retrieves the corresponding data of matrix B from the weight memory 502 and caches it in each PE of the arithmetic circuit. The arithmetic circuit retrieves the data of matrix A from the input memory 501 and performs matrix operations with matrix B. The partial result or the final result of the obtained matrix is ​​stored in the accumulator 508.

[0191] The vector computation unit 507 can further process the output of the arithmetic circuit, such as vector multiplication, vector addition, exponentiation, logarithmic operations, size comparisons, etc. For example, the vector computation unit 507 can be used for network computation in non-convolutional / non-FC layers of neural networks, such as pooling, batch normalization, local response normalization, etc.

[0192] In some implementations, the vector computation unit 507 can store the processed output vector into a unified buffer 506. For example, the vector computation unit 507 can apply a nonlinear function to the output of the arithmetic circuit 503, such as a vector of accumulated values, to generate activation values. In some implementations, the vector computation unit 507 generates normalized values, merged values, or both. In some implementations, the processed output vector can be used as activation input to the arithmetic circuit 503, for example, for use in subsequent layers of a neural network.

[0193] The unified memory 506 is used to store input data and output data.

[0194] The weight data is directly accessed through the memory access controller 505 (DMAC) to move input data from external memory to input memory 501 and / or unified memory 506, store weight data from external memory into weight memory 502, and store data from unified memory 506 into external memory.

[0195] The bus interface unit (BIU) 510 is used to enable interaction between the main CPU, DMAC and instruction fetch memory 509 via a bus.

[0196] The instruction fetch buffer 509, connected to the controller 504, is used to store the instructions used by the controller 504.

[0197] The controller 504 is used to call the instructions cached in the instruction memory 509 to control the operation of the computing accelerator.

[0198] Generally, the unified memory 506, input memory 501, weight memory 502, and instruction fetch memory 509 are all on-chip memories, while the external memory is memory outside the NPU. This external memory can be double data rate synchronous dynamic random access memory (DDR SDRAM), high bandwidth memory (HBM), or other readable and writable memory.

[0199] in, Figure 3 and Figure 4 The operations of each layer in the convolutional neural network shown can be performed by the operation circuit 503 or the vector calculation unit 507.

[0200] The above-mentioned Figure 2 The execution device 110 in the embodiment is capable of executing each step of the image processing method of this application. Figure 3 and Figure 4 The CNN model shown and Figure 5 The chip shown can also be used to execute various steps of the image processing method of the embodiments of this application. The method for constructing a neural network and the image processing method of the embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0201] like Figure 6As shown, this application embodiment provides a system architecture 300. The system architecture includes a local device 301, a local device 302, an execution device 210, and a data storage system 250, wherein the local devices 301 and 302 are connected to the execution device 210 through a communication network.

[0202] The execution device 210 can be implemented by one or more servers. Optionally, the execution device 210 can be used in conjunction with other computing devices, such as data storage devices, routers, load balancers, etc. The execution device 210 can be deployed on a single physical site or distributed across multiple physical sites. The execution device 210 can use data in the data storage system 250 or call program code in the data storage system 250 to implement the method for constructing a neural network according to the embodiments of this application.

[0203] Specifically, the execution device 210 can perform the following process: constructing a search space according to the application requirements of the target neural network; clustering multiple elements in the search space to obtain clustering results; and selecting a target network structure that meets preset requirements from multiple network structures indicated by the multiple elements according to the clustering results; and building the target neural network according to the target network structure.

[0204] The device 210 can build a target neural network through the above process, which can be used for image classification or image processing, etc.

[0205] Users can interact with execution device 210 by operating their respective user devices (e.g., local device 301 and local device 302). Each local device can represent any computing device, such as a personal computer, computer workstation, smartphone, tablet, smart camera, smart car or other type of cellular phone, media consumption device, wearable device, set-top box, game console, etc.

[0206] Each user's local device can interact with the execution device 210 through a communication network of any communication mechanism / standard. The communication network can be a wide area network, a local area network, a point-to-point connection, or any combination thereof.

[0207] In one implementation, local devices 301 and 302 obtain relevant parameters of the target neural network from execution device 210, deploy the target neural network on local devices 301 and 302, and use the target neural network for image classification or image processing, etc.

[0208] In another implementation, the target neural network can be directly deployed on the execution device 210. The execution device 210 obtains the image to be processed from the local devices 301 and 302, and performs classification or other types of image processing on the image to be processed according to the target neural network.

[0209] The aforementioned execution device 210 can also be a cloud device, in which case the execution device 210 can be deployed in the cloud; or, the aforementioned execution device 210 can also be a terminal device, in which case the execution device 210 can be deployed on the user terminal side. This application embodiment does not limit this.

[0210] Let's combine the following... Figure 7 The method 700 for constructing a neural network according to embodiments of this application will be described in detail. Figure 7 The method shown can be executed by a device for building a neural network, which can be a mobile terminal, computer, server, or other device with sufficient computing power to build a neural network.

[0211] Figure 7 The method shown includes steps 710, 720 and 730, which are described in detail below.

[0212] S710 constructs the search space based on the application requirements of the target neural network.

[0213] The search space may include M elements, which can be used to indicate M network structures. Each of the M elements may include the number of stages in the corresponding network structure, the number of blocks in the stage, and / or the number of channels in the block, where M is a positive integer.

[0214] Optionally, each of the M elements including the corresponding network structure refers to the network structure indicated by each of the M elements.

[0215] Accordingly, for each of the M elements, each element includes the number of stages in the network structure indicated by the element, the number of blocks in the stage, and / or the number of channels in the block.

[0216] The target neural network described above can be a deep neural network. For ease of explanation, a convolutional neural network is used as an example in this application.

[0217] In the embodiments of this application, the network structure may include some stages in a convolutional neural network. For example, the network structure may refer to the part of the convolutional neural network that the user wants to adjust (or the part to be adjusted). Therefore, the network structure may also be called the network structure to be adjusted.

[0218] It should be noted that, in the embodiments of this application, network structure precision may refer to the precision (or accuracy) of the neural network constructed from the network structure.

[0219] The target neural network described above can be constructed from at least one of the network structures, which may include one or more stages, and each stage may include at least one block.

[0220] A block can be composed of basic atomic units in a convolutional neural network, including convolutional layers, pooling layers, fully connected layers, and nonlinear activation layers.

[0221] In convolutional neural networks, features typically exist in three dimensions (length, width, and depth). A feature can be viewed as a superposition of multiple two-dimensional features, where each two-dimensional feature can be called a feature map. Alternatively, a feature map (two-dimensional feature) can also be called a channel of the feature.

[0222] The length and width of a feature map can also be referred to as the resolution of the feature map.

[0223] Optionally, the number of blocks in different stages can be different.

[0224] Similarly, the resolution of the input feature map and the resolution of the output feature map can be different at different stages.

[0225] Optionally, the number of channels in different blocks can be different. It should be understood that the number of channels in a block can also be referred to as the width of the block, and this embodiment of the application does not limit this.

[0226] Similarly, the resolution of the input feature map and the resolution of the output feature map can also be different for different blocks.

[0227] In embodiments of this application, the elements may further include the resolution of the input feature map of the stage, the resolution of the output feature map of the stage, the resolution of the input feature map of the block, and / or the resolution of the output feature map of the block.

[0228] As the parameters (of the convolutional neural network) included in the elements differ, they will directly affect the performance and speed of the convolutional neural network. Therefore, in this embodiment of the application, a search space can be constructed for these parameters based on the application requirements of the target neural network, and a target network structure that meets the preset requirements can be selected in the search space.

[0229] In this application embodiment, the application requirements of the target neural network may include the running speed of the target neural network, the number of parameters of the target neural network, or the structural requirements of the target neural network, wherein the structural requirements include the number of stages in the target neural network structure, the number of blocks in each stage, and / or the number of channels in each block.

[0230] Optionally, the search space can be constructed based on the operating speed of the target neural network and / or the number of parameters of the target neural network. The search space may include all possible network structures that meet the application requirements of the target neural network.

[0231] In this embodiment of the application, a search space is constructed based on the running speed and / or the number of parameters of the target neural network. This allows for the filtering out of some low-performance network structures during the search space construction process, thereby improving the efficiency of constructing the neural network.

[0232] Optionally, the application requirements of the target neural network may also include the structural requirements of the network structure.

[0233] Optionally, the structural requirements of the network structure may include a requirement for the number of stages in the network structure. This requirement can be a range of values ​​for the number of stages; for example, the allowed number of stages in the network structure is between 2 and 5. Alternatively, the requirement can also be a specific number of stages; for example, the number of stages in the network structure is 4.

[0234] Similarly, the structural requirements of the network structure may also include the number of blocks in each stage and / or the number of channels in the blocks.

[0235] At this point, the search space can be constructed based on the running speed of the target neural network, the number of parameters of the target neural network, the required number of stages in the network structure, the required number of blocks in each stage, and / or the required number of channels in each block. The search space can include all possible network structures that meet the application requirements of the target neural network.

[0236] In this application, a search space can be constructed based on existing neural networks, and a target network structure that meets preset requirements can be selected in the search space.

[0237] For example, in this embodiment of the application, a search space can be constructed based on existing ResNets, and a target network structure that meets preset requirements can be selected in the search space.

[0238] Optionally, ResNets can divide the neural network into six stages based on the resolution of the output feature maps. The first stage contains only one convolutional operation, and the last stage performs probability regression based on a fully connected operation. The second through fifth stages are each composed of one or more stacked residual blocks, the structure of which can be as follows: Figure 8 As shown.

[0239] The specific descriptions of the ResNet network structure and residual blocks can be found in existing technologies, and will not be repeated in the embodiments of this application.

[0240] For example, in the embodiments of this application, the second to fifth stages of the ResNets described above can be used as the network structure to be adjusted, the search space is constructed based on the network structure, and it is stipulated that the number of channels in the blocks in each stage is the same.

[0241] It should be noted that the number of stages and the number of channels in the blocks in this application embodiment are merely examples and not limitations. In practice, it is not limited to the number of stages in the network structure to be adjusted being 4, nor is it limited to the number of channels in the blocks in each stage being the same. The structural requirements of the network structure can be determined according to specific application needs, and this application embodiment does not limit this.

[0242] As an example, and not a limitation, after determining that the number of stages in the network structure to be adjusted is 4 (i.e., the second to fifth stages of ResNets), it is actually only necessary to adjust the number of blocks and the number of channels in these 4 stages. An octet can be used to encode the network structure. For ease of description, this octet can be called the block connection style code (BCSC). Specifically, a network structure can be encoded as follows:

[0243] BCSC={D2, D3, D4, D5, C2, C3, C4, C5}

[0244] Where D2 represents the number of blocks in the second stage, D3 represents the number of blocks in the third stage, D4 ​​represents the number of blocks in the fourth stage, D5 represents the number of blocks in the fifth stage, C2 represents the number of channels in the blocks in the second stage, C3 represents the number of channels in the blocks in the second stage, C4 represents the number of channels in the blocks in the second stage, and C5 represents the number of channels in the blocks in the second stage.

[0245] It should be understood that the above-described BCSC (octet) encoding method is merely an example and not a limitation. Other existing encoding methods can also be used to encode the network structure in the embodiments of this application.

[0246] In this embodiment of the application, the M elements in the search space can be M BCSCs, where each BCSC can indicate a network structure.

[0247] In one possible implementation, all M possible network structures can be determined according to the application requirements of the target neural network; and each network structure can be encoded using BCSC to obtain M BCSCs; at this time, these M BCSCs can be used as M elements to form the search space.

[0248] In another possible implementation, based on the application requirements of the target neural network, there can be N possible network structures. Correspondingly, an initial search space including N BCSCs can be obtained. At this point, the initial search space can be filtered to remove low-performance network structures, resulting in a search space including M elements. See the following for details. Figure 9 The description of method 900 in the text.

[0249] Optionally, an initial search space can be constructed according to the application requirements of the target neural network. The initial search space includes N initial elements, which are used to indicate N initial network structures. Each of the N initial elements includes the number of stages in the corresponding initial network structure, the number of blocks in the stage, and / or the number of channels in the block, where N is a positive integer greater than or equal to M. The N initial network structures indicated by the N initial elements can be filtered according to preset rules to obtain the M elements in the search space.

[0250] The preset rules may include:

[0251] If the number of blocks in each stage of the first initial network structure indicated by the first initial element among the N initial elements is not greater than the number of blocks in the corresponding stage of the second initial network structure indicated by the second initial element among the N initial elements, and the number of channels in each block of each stage of the first initial network structure is not greater than the number of channels in each block of the corresponding stage of the second initial network structure, then the first initial element is deleted from the initial search space.

[0252] As can be seen from the above preset rules, if performing the following operations on the first initial network structure several times results in a network structure identical to the second initial network structure, it indicates that the accuracy of the second initial network structure is higher than that of the first initial network structure. Accordingly, the first initial element can be deleted from the initial search space. Specific operations include:

[0253] (1) Copy a block in a stage;

[0254] (2) Increase the number of channels (or width) of a block in a stage.

[0255] For example, Figure 10 N network structures are shown, where N is a positive integer greater than 1. In network structure 1, stage 1 includes 1 block with 64 channels; stage 2 includes 2 blocks with 128 channels each; stage 3 includes 2 blocks with 256 channels each; and stage 4 includes 4 blocks with 512 channels each. In network structure 2, stage 1 includes 1 block with 64 channels; stage 2 includes 1 block with 128 channels each; stage 3 includes 2 blocks with 256 channels each; and stage 4 includes 3 blocks with 512 channels each.

[0256] from Figure 10 As can be seen, copying a block from stage 2 and a block from stage 4 of network structure 2 results in stage 2 containing two blocks with 128 channels and stage 4 containing four blocks with 512 channels. At this point, the copied network structure 2 has the same network structure as network structure 1, indicating that the accuracy of network structure 2 is lower than that of network structure 1. Therefore, network structure 2 (or the element corresponding to network structure 2) can be deleted from the initial search space.

[0257] Alternatively, you can directly compare two initial elements in the initial search space.

[0258] For example, the BCSC of the first initial element and the BCSC of the second initial element can be directly compared. If each element in the BCSC of the first initial element is smaller than the corresponding element in the BCSC of the second initial element, it indicates that the accuracy of the first initial network structure is lower than that of the second initial network structure. Accordingly, the first initial element can be deleted from the initial search space.

[0259] In this embodiment of the application, according to the above-mentioned preset rules, without training the initial network structure indicated by the N initial elements in the initial search space, the low-performance network structure among the N initial network structures can be screened out based solely on the structural information of the N initial network structures. Therefore, the efficiency of constructing a neural network can be improved.

[0260] S720, Select the target network structure from the M network structures based on the distribution relationship of the unevaluated elements in the search space.

[0261] The distribution relationship of the unevaluated elements can be the clustering result of the unevaluated elements.

[0262] In this embodiment, based on the distribution relationship of the unevaluated elements, representative elements can be easily selected from the unevaluated elements. Then, based on the representative elements, the target network structure can be efficiently selected from the M network structures, thereby enabling the efficient construction of a neural network that meets the performance requirements.

[0263] Optionally, a candidate set can be determined based on the distribution relationship of the unevaluated elements, which may include representative elements among the selected unevaluated elements.

[0264] It should be noted that the methods for determining the candidate set described below are merely examples and not limitations. In this application embodiment, other methods can also be used to determine the candidate set. This application embodiment does not limit the specific method for determining the candidate set based on the distribution relationship of unevaluated elements.

[0265] For example, in the embodiments of this application, a candidate set (including representative elements among the unevaluated elements) can be obtained based on the distribution relationship of the unevaluated elements in several ways.

[0266] Method 1:

[0267] A candidate set (of unevaluated elements) can be constructed such that the distance between elements in the candidate set is as large as possible. This makes the distribution of elements in the set as dispersed as possible among the unevaluated elements, thus enabling the selection of representative elements from the unevaluated elements.

[0268] For example, the candidate set might be obtained through the following method:

[0269] (1) Initialize each set, where the set is a platform. An empty set, a set Let T be the set of all unevaluated elements, where T is a positive integer;

[0270] (2) From the platform Randomly select an element β from the set and add element β to the set. At the same time, from the platform Remove element β;

[0271] (3) For the platform Any element β in i Calculate the arbitrary element β i With platform The minimum distance between elements in the array is denoted as dis. i ;

[0272] (4) Select set {dis i The minimum value in the sequence |i = 1, 2, 3, ... is denoted as dis. j , the dis jcorresponding set The element in is denoted as element β. j ;

[0273] (5) Element β j Add to collection Simultaneously from the set Remove element β j ;

[0274] (6) If the platform If the number of elements is greater than T, then execute step (7); otherwise, execute step (3) for iteration.

[0275] (7) Set As a candidate set, it can be subsequently used from the set. Select one or more elements to evaluate.

[0276] Method 2:

[0277] A candidate set of unevaluated elements can be constructed such that each element in the candidate set is located at the center of a subset of the unevaluated elements (where each element is located).

[0278] For example, the candidate set might be obtained through the following method:

[0279] (1) Initialize each set, where the set is a platform. An empty set, a set Let K be the set of all unevaluated elements, where K is a positive integer;

[0280] (2) For sets Any element β in i Calculate the arbitrary element β i At the market K-neighbors (i.e., the set) In and element β i The K closest elements are denoted as Where i and K are positive integers;

[0281] (3) For sets Any element β in i Calculate the arbitrary element β i and The minimum distance between elements in the array is denoted as dis. i ;

[0282] (4) Select set {dis i The minimum value in the sequence |i = 1, 2, 3, ...} is denoted as dis. j , the dis j The corresponding set The element in is denoted as element β. j ;

[0283] (5) Element β j Add to collection Simultaneously from the set Remove

[0284] (6) If the set If the number of elements is less than K, then execute step (7); otherwise, execute step (2) for iteration.

[0285] (7) Set As a candidate set, it can be subsequently selected from the platform. Select one or more elements to evaluate.

[0286] Method 3:

[0287] Unevaluated elements can be clustered to obtain clustering results (the clustering results of unevaluated elements can also be considered as the distribution relationship of unevaluated elements), and representative elements among the unevaluated elements can be selected based on the clustering results of unevaluated elements, i.e., candidate set.

[0288] For ease of understanding, the following embodiments will be described using the example of selecting a candidate set by clustering (of unevaluated elements).

[0289] Optionally, the unevaluated elements in the search space can be clustered to obtain the clustering results of the unevaluated elements, and then the target network structure can be selected from the M network structures based on the clustering results of the unevaluated elements.

[0290] In this embodiment of the application, before clustering the M elements, the M elements (i.e., the M BCSCs) can be standardized first. The specific standardization process can refer to the prior art, and this application does not limit it.

[0291] Optionally, the mean and variance of the M BCSCs can be calculated in each dimension.

[0292] For example, the mean of each dimension can be denoted as... in, This represents the average number of blocks in the second stage. This represents the average number of blocks in the third stage. This represents the average number of blocks in the fourth stage. This represents the average number of blocks in the fifth stage. This represents the average number of channels in the blocks during the second stage. This represents the average number of channels in the blocks during the third stage. This represents the average number of channels in the blocks during the fourth stage. This represents the average number of channels in the blocks during the fifth stage.

[0293] Similarly, the variance of each dimension can be denoted as... in, This represents the variance of the number of blocks in the second stage. This represents the variance of the number of blocks in the third stage. This represents the variance of the number of blocks in the fourth stage. This represents the variance of the number of blocks in the fifth stage. This represents the variance of the number of channels in a block during the second stage. This represents the variance of the number of channels in a block during the third stage. This represents the variance of the number of channels in the blocks during the fourth stage. This represents the variance of the number of channels in the blocks in the fifth stage.

[0294] For each BCSC = {D2, D3, D4, D5, C2, C3, C4, C5} above, after standardization, we can obtain... in, This represents the result after standardizing the BCSC.

[0295] At this point, the M items obtained after standardization can be... Clustering is performed. This application does not limit the clustering method used; specific clustering methods can be found in existing technologies, and will not be elaborated here.

[0296] For example, the K-means algorithm can be used to process the M standardized values. Perform clustering.

[0297] Optionally, K elements can be determined from the unevaluated elements based on their distribution relationship, where K is a positive integer less than M; and a target network structure can be selected from the M network structures based on the K elements.

[0298] Clustering the M elements (i.e., the M BCSCs) can group the M elements into one or more clusters.

[0299] Accordingly, the clustering result of the M elements may include one or more clusters obtained after clustering the M elements. Elements within the same cluster have a stronger similarity to elements in other clusters.

[0300] Optionally, the clustering result may also include the center of each cluster, which can be considered as the most representative element in that cluster.

[0301] For example, the clustering result of the unevaluated elements may include K clusters, where K is a positive integer less than M. In this case, the K elements among the unevaluated elements determined according to the distribution relationship of the unevaluated elements may be the K elements corresponding to the cluster centers of the K clusters.

[0302] In this embodiment of the application, clustering the M elements in the search space can group the network structures with similar structures indicated by the M elements into a cluster. Based on the multiple clusters obtained by clustering, it is helpful to efficiently select the target network structure that meets the preset requirements from the M network structures.

[0303] For example, the clustering result may include K clusters. In this case, an element can be selected from each cluster to obtain the above K elements, and the K network structures indicated by the K elements can be evaluated to obtain the evaluation results of the K network structures.

[0304] The evaluation of the K network structures indicated by the K elements can refer to testing the accuracy (or precision) of the K network structures.

[0305] Furthermore, the K elements may be the K elements corresponding to the centers of the K clusters.

[0306] Optionally, selecting the target network structure from the M network structures based on the K elements may include:

[0307] The K network structures indicated by the K elements in the unevaluated elements are evaluated to obtain the evaluation results of the evaluated elements, which include the evaluation results of the K network structures; based on the evaluation results of the evaluated elements, a target network structure is selected from the M network structures.

[0308] In this embodiment of the application, the K network structures indicated by the K elements corresponding to the centers of the K clusters are the most representative network structures in the search space. Based on the evaluation results of the K network structures indicated by the K elements corresponding to the centers of the K clusters, a target network structure that meets the preset requirements is selected from the M network structures. This can efficiently select the target network structure that meets the preset requirements from the M network structures, thereby improving the efficiency of building neural networks.

[0309] If at least one of the K network structures indicated by the K elements has an accuracy (or precision) that meets a preset requirement, then that network structure is the target network structure, and a neural network can be constructed based on that network structure.

[0310] If no network structure meets the preset requirements, it is necessary to continue iteratively selecting from the M network structures based on the evaluation results of the evaluated network structures until a target network structure that meets the preset requirements is selected.

[0311] For example, a model can be created for the first unevaluated element based on the evaluation results of the evaluated elements, and the model of the first unevaluated element can be obtained. The first unevaluated element includes other elements in the search space besides the evaluated elements. Based on the model of the first unevaluated element, a target network structure can be selected from the M network structures.

[0312] Alternatively, modeling can be performed using the following methods:

[0313] Method 1:

[0314] Based on the evaluation results of the evaluated elements and the following formula, a model can be built for the first unevaluated element to obtain the model of the unevaluated network structure:

[0315] and / or (g(x)-l(x)) 2 ;

[0316] Where x represents the unevaluated network structure in the search space, y * Let g(x) represent the precision threshold, and g(x) represent precision greater than the precision threshold y. * The conditional probability density function of the network structure x. l(x) represents a precision less than or equal to the precision threshold y. * The conditional probability density function of the network structure x. G * (x|x i ) represents a mixture Gaussian distribution. κ(x, x) i ) represents x and x i The Euclidean distance between them, κ*(x, x i ) represents the expression by κ(x, x i The distance function composed of ) Let y represent the i-th network structure in the evaluated network structures. i ω represents the accuracy of the i-th network structure among the evaluated network structures. i G represents * (x|x i The corresponding weights, Z represents the normalization factor, and σ represents G. * (x|x i ) is a hyperparameter of ), where i is a positive integer and e is the base of the natural logarithm function.

[0317] In the above model, G * (x|x i The corresponding weight ω i It can satisfy: ω i >0, And when y i ≤y * At that time, y i The larger ω is i The smaller, when y i >y * At that time, y i The larger ω is i The larger; Z can satisfy The value of σ can be half the average Euclidean distance between the elements corresponding to the centers of each cluster after clustering.

[0318] Optionally, the above It is positively correlated with the performance of the network structure, that is to say, The larger the value, the better. The better the performance of the corresponding network structure. (g(x)-l(x)) 2 It is positively correlated with the uncertainty of network structure performance, that is, (g(x)-l(x)) 2 The larger the value of (g(x)-l(x)) is, the better. 2 The higher the uncertainty of the corresponding network structure performance.

[0319] Optionally, while using and (g(x)-l(x)) 2 In such cases, the ratio of the two models can be set based on experience.

[0320] For example, a 3:1 ratio can be used. and (g(x)-l(x)) 2 Modeling is performed.

[0321] At this point, if it is necessary to select four network structures, then one of them can be selected. The three network structures with the largest values, and (g(x)-l(x)) 2 The network structure with the largest value.

[0322] If at least one of these four network structures meets the preset accuracy (or precision) requirement, then that network structure is the target network structure, and a neural network can be constructed based on that network structure.

[0323] If no network structure meets the preset requirements, then a target network structure is selected from the M network structures based on the evaluation results of the previously evaluated network structures (including the four network structures just selected above).

[0324] Method 2:

[0325] Based on the evaluation results of the evaluated elements and the following formula, a model can be built for the first unevaluated element to obtain the model of the first unevaluated element:

[0326]

[0327] Where x represents an unevaluated element in the search space; x i Let y represent the i-th element among the evaluated elements. i This represents the precision of the i-th element among the evaluated elements; τ = max(y i ), Let f(x) represent the expectation function; f(x) is a random variable following a Gaussian distribution, and the mean μ(x) and variance σ(x) of f(x) are related to... The input x satisfies the following relationship:

[0328] μ(x)=k T (K+η 2 I) -1 Y,

[0329] σ(x)=1-k T (K+η 2 I) -1 k.

[0330] Where n is the number of network structures evaluated, and Y is y i The vector formed, Y∈R n Y i =y i k is κ(x, x i A vector consisting of k ∈ R n k i =κ(x, x) i ), K is κ(x) i x j The matrix formed by K∈R is given by ( ). n×n K i，j =κ(x) i x j ), σ is a hyperparameter, e is the base of the natural logarithm function; I is the identity matrix, η is also a hyperparameter, and i and j are positive integers.

[0331] It should be noted that the above modeling method is only an example and not a limitation. Other methods can also be used for modeling in this application embodiment.

[0332] Furthermore, if the target network structure is not selected from the M network structures based on the distribution relationship of unevaluated elements in the search space, the method may further include:

[0333] Based on the distribution relationship of the first unevaluated element and the model of the first unevaluated element constructed based on the evaluated element, a target network structure is selected from the M network structures.

[0334] Optionally, selecting the target network structure from the M network structures based on the distribution relationship of the first unevaluated elements and the model of the first unevaluated elements constructed based on the evaluated elements may include:

[0335] Based on the distribution relationship of the first unevaluated elements, determine L elements from the first unevaluated elements, where L is a positive integer less than M; select a target network structure from the M network structures based on the L elements and the model of the first unevaluated elements constructed based on the evaluated elements.

[0336] Optionally, the step of selecting the target network structure from the M network structures based on the L elements and the model constructed based on the evaluated elements may include:

[0337] Based on the model of the first unevaluated element constructed from the evaluated elements, Q elements are determined from the L elements, where Q is a positive integer less than L; the Q network structures indicated by the Q elements are evaluated to obtain the evaluation results of the first evaluated element, which includes the evaluation results of the K network structures and the evaluation results of the Q network structures; based on the distribution relationship between the evaluation results of the first evaluated element and the second unevaluated element, a target network structure is selected from the M network structures, where the second unevaluated element includes other elements in the search space besides the first evaluated element.

[0338] Wherein, the distribution relationship of the first unevaluated element can be the clustering result of the first unevaluated element, and the L elements can be elements in the L clusters included in the clustering result of the first unevaluated element.

[0339] Optionally, the L elements can be the L elements corresponding to the centers of the L clusters.

[0340] Furthermore, if no target network structure is selected from the M network structures based on the distribution relationship of the first unevaluated elements and the model of the first unevaluated elements constructed based on the evaluated elements, the method may further include:

[0341] Based on the distribution relationship of the second unevaluated element and the model of the second unevaluated element constructed based on the first evaluated element, a target network structure is reselected from the M network structures.

[0342] It should be noted that the method described above for reselecting a target network structure from the M network structures based on the distribution relationship of the second unevaluated elements and the model of the second unevaluated elements constructed based on the first evaluated elements is similar to the method described above for selecting a target network structure from the M network structures based on the distribution relationship of the first unevaluated elements and the model of the first unevaluated elements constructed based on the evaluated elements, and will not be repeated here.

[0343] This process will continue iteratively until a target network structure that meets the preset requirements is selected.

[0344] Figure 9 A schematic flowchart of an image processing method 900 provided in an embodiment of this application is shown. The method can be executed by a device or equipment capable of image processing, such as a terminal device, computer, server, etc.

[0345] S910 constructs the initial search space based on the application requirements of the target neural network.

[0346] The initial search space may include N initial elements, which can be used to indicate N network structures. Each of the M elements may include the number of stages in the corresponding network structure, the number of blocks in the stage, and / or the number of channels in the block, where N is a positive integer.

[0347] Optionally, the target neural network may be a convolutional neural network.

[0348] In the embodiments of this application, the network structure may include some stages in a convolutional neural network. For example, the network structure may refer to the part of the convolutional neural network that the user wants to adjust (or the part to be adjusted). Therefore, the network structure may also be called the network structure to be adjusted.

[0349] The target neural network described above can be constructed from at least one of the network structures, which may include one or more stages, and each stage may include at least one block.

[0350] S920 filters the initial search space based on preset rules to obtain the search space.

[0351] The search space may include M elements, which can be used to indicate M network structures. The elements may include the number of stages in the network structure, the number of blocks in the stage, and / or the number of channels in the block, where M is a positive integer less than or equal to N.

[0352] Optionally, the preset rules may include:

[0353] If the number of blocks in each stage of the first initial network structure indicated by the first initial element among the N initial elements is not greater than the number of blocks in the corresponding stage of the second initial network structure indicated by the second initial element among the N initial elements, and the number of channels in each block of each stage of the first initial network structure is not greater than the number of channels in each block of the corresponding stage of the second initial network structure, then the first initial element is deleted from the initial search space.

[0354] As can be seen from the above preset rules, if performing the following operations on the first initial network structure several times results in a network structure identical to the second initial network structure, it indicates that the accuracy of the second initial network structure is higher than that of the first initial network structure. Accordingly, the first initial element can be deleted from the initial search space. Specific operations include:

[0355] (1) Copy a block in a stage;

[0356] (2) Increase the number of channels (or width) of a block in a stage.

[0357] For example, Figure 10 N network structures are shown, where N is a positive integer greater than 1. In network structure 1, stage 1 includes 1 block with 64 channels; stage 2 includes 2 blocks with 128 channels each; stage 3 includes 2 blocks with 256 channels each; and stage 4 includes 4 blocks with 512 channels each. In network structure 2, stage 1 includes 1 block with 64 channels; stage 2 includes 1 block with 128 channels each; stage 3 includes 2 blocks with 256 channels each; and stage 4 includes 3 blocks with 512 channels each.

[0358] from Figure 10As can be seen, copying a block from stage 2 and a block from stage 4 of network structure 2 results in stage 2 containing two blocks with 128 channels and stage 4 containing four blocks with 512 channels. At this point, the copied network structure 2 has the same network structure as network structure 1, indicating that the accuracy of network structure 2 is lower than that of network structure 1. Therefore, network structure 2 (or the element corresponding to network structure 2) can be deleted from the initial search space.

[0359] Alternatively, you can directly compare two initial elements in the initial search space.

[0360] For example, the BCSC of the first initial element and the BCSC of the second initial element can be directly compared. If each element in the BCSC of the first initial element is smaller than the corresponding element in the BCSC of the second initial element, it indicates that the accuracy of the first initial network structure is lower than that of the second initial network structure. Accordingly, the first initial element can be deleted from the initial search space.

[0361] S930 clusters the unevaluated elements in the search space to obtain K clusters.

[0362] Elements within the same cluster exhibit stronger similarity to elements in other clusters. The clustering result may also include the center of each cluster, which can be considered the most representative element within that cluster.

[0363] In this embodiment of the application, the unevaluated elements in the search space are M elements in the search space. Before clustering the M elements, the M elements (i.e. M BCSCs) can be standardized first. The specific standardization process can refer to the prior art, and this application does not limit it.

[0364] Optionally, the mean and variance of the M BCSCs can be calculated in each dimension.

[0365] For example, the mean of each dimension can be denoted as... in, This represents the average number of blocks in the second stage. This represents the average number of blocks in the third stage. This represents the average number of blocks in the fourth stage. This represents the average number of blocks in the fifth stage. This represents the average number of channels in the blocks during the second stage. This represents the average number of channels in the blocks during the third stage. This represents the average number of channels in the blocks during the fourth stage. This represents the average number of channels in the blocks during the fifth stage.

[0366] Similarly, the variance of each dimension can be denoted as... in, This represents the variance of the number of blocks in the second stage. This represents the variance of the number of blocks in the third stage. This represents the variance of the number of blocks in the fourth stage. This represents the variance of the number of blocks in the fifth stage. This represents the variance of the number of channels in a block during the second stage. This represents the variance of the number of channels in a block during the third stage. This represents the variance of the number of channels in the blocks during the fourth stage. This represents the variance of the number of channels in the blocks in the fifth stage.

[0367] For each BCSC = {D2, D3, D4, D5, C2, C3, C4, C5} above, after standardization, we can obtain... in, This represents the result after standardizing the BCSC.

[0368] At this point, the M items obtained after standardization can be... Clustering is performed. This application does not limit the clustering method used; specific clustering methods can be found in existing technologies, and will not be elaborated here.

[0369] For example, the K-means algorithm can be used to process the M standardized values. Perform clustering.

[0370] S940, evaluate the K network structures indicated by the elements corresponding to the centers of the K clusters, and obtain the evaluation results of the K network structures.

[0371] If at least one of the K network structures indicated by the elements corresponding to the centers of the K clusters has an accuracy (or precision) that meets a preset requirement, then that network structure is the target network structure, and a neural network can be constructed based on that network structure.

[0372] If no network structure meets the preset requirements, then continue executing S950.

[0373] S950: Based on the evaluation results of the evaluated network structures, model the unevaluated network structures in the search space to obtain the model of the unevaluated network structures.

[0374] Optionally, based on the evaluation results of the evaluated network structures, models can be built for the unevaluated network structures among the M network structures to obtain models of the unevaluated network structures. The evaluated network structures may include the K network structures indicated by the elements corresponding to the centers of the aforementioned K clusters.

[0375] It should be understood that the embodiments of this application do not limit the specific modeling method. For a description of the modeling method, please refer to the above. Figure 7 The embodiments in Method 700 will not be described in detail here.

[0376] S960 clusters the unevaluated elements in the search space to obtain P clusters.

[0377] Elements within the same cluster exhibit stronger similarity to elements in other clusters. The clustering result may also include the center of each cluster, which can be considered the most representative element within that cluster.

[0378] In this embodiment of the application, during the first iteration, the unevaluated elements in the search space are MK elements in the search space; during the second iteration, the unevaluated elements in the search space are M-2*K elements in the search space; and in subsequent iterations, the unevaluated elements in the search space are M elements minus the evaluated elements in the search space.

[0379] It should be noted that the embodiments of this application do not limit the evaluation of the same number of elements in each iteration. The above K elements are only examples and not limitations. That is to say, the number of elements evaluated in each iteration can be different.

[0380] S970, based on the model of the unevaluated network structure, select K clusters from the P clusters.

[0381] Optionally, the accuracy of the unevaluated network structure can be evaluated based on the model of the unevaluated network structure to obtain the evaluation accuracy; then, K network structures with the highest accuracy can be selected from the P clusters for evaluation.

[0382] It should be noted that if no network structure meets the preset requirements, the above steps S940, S950, S960, and S970 can be performed iteratively, that is, continue to iteratively select in the search space until a target network structure that meets the preset requirements is selected.

[0383] In subsequent iterations, the unevaluated elements in the search space are always elements other than those that have been evaluated.

[0384] Optionally, during the first iteration, the unevaluated elements in the search space are all elements in the search space other than the K elements that have been evaluated, that is, there are MK unevaluated elements at this time.

[0385] The subsequent iterations are similar to the first iteration, and will not be described in detail here.

[0386] For example, during the first iteration, the unevaluated elements in the search space can be clustered into P clusters according to the method in S960 above. At this time, K elements can be selected from the P clusters according to the model of the unevaluated elements, and these K elements can be evaluated according to the method in S940 above, where P and K are positive integers.

[0387] If at least one of the K network structures has an accuracy (or precision) that meets the preset requirements, then that network structure is the target network structure, and a neural network can be constructed based on that network structure.

[0388] If no network structure meets the preset requirements, the method in S950 above is used to model the unevaluated elements based on the evaluation results of the 2*K network structures that have been evaluated. Then, the method in S960 and S970 above is used to select a target neural network that meets the preset requirements in the search space based on the model of the unevaluated elements and the distribution relationship between the unevaluated elements and the unevaluated elements.

[0389] At this point, if at least one network structure has an accuracy (or precision) that meets the preset requirements, then that network structure is the target network structure, and a neural network can be constructed based on that network structure.

[0390] If a network structure that still does not meet the preset requirements is not found, iterative selection can continue according to the above steps S940, S950, S960, and S970 until a target network structure that meets the preset requirements is selected.

[0391] Figure 11 A schematic flowchart of an image processing method 1100 provided in an embodiment of this application is shown. The method can be executed by a device or equipment capable of image processing, such as a terminal device, computer, server, etc.

[0392] Figure 11 The target neural network used in the image processing method 1100 can be obtained through the above... Figure 7 Method 700 or Figure 9 The method described in 900 can be used to construct the structure, but it can also be constructed using other methods. This application does not limit the specific method used in this embodiment.

[0393] S1110, Obtain the image to be processed.

[0394] The image to be processed may be an image captured by a camera of a terminal device (or a computer, server, or other device or equipment), or it may be an image obtained from within the terminal device (or a computer, server, or other device or equipment) (e.g., an image stored in the terminal device's photo album, or an image obtained by the terminal device from the cloud). This application embodiment does not limit this.

[0395] S1120, The image to be processed is classified according to the target neural network to obtain the classification result of the image to be processed.

[0396] The target neural network is constructed from a target network structure, which is constructed using method 700 in Figure 7 above. Figure 9 The method was obtained using method 900.

[0397] It should be understood that the target neural network used in the above image processing method needs to be trained again based on the training images before it can perform image classification. The trained target neural network can then classify the images to be processed.

[0398] In other words, the above methods can be used. Figure 7 Method 700 or Figure 9 The target neural network is constructed using the target network structure obtained by method 900. Next, the target neural network is trained based on the training images. After training, the target neural network can be used to classify the images to be processed.

[0399] In this application, since the target neural network is constructed using the above-mentioned... Figure 7 Method 700 or Figure 9 The target network structure obtained by method 900 is more in line with or close to the application requirements of neural networks. Using such a neural network for image classification can achieve better image classification results (e.g., more accurate classification results, etc.).

[0400] Figure 12 This is a schematic diagram of the hardware structure of the apparatus for constructing a neural network provided in an embodiment of this application. Figure 12 The apparatus 3000 for constructing a neural network shown (specifically, the apparatus 3000 can be a computer device) includes a memory 3001, a processor 3002, a communication interface 3003, and a bus 3004. The memory 3001, processor 3002, and communication interface 3003 are interconnected via the bus 3004.

[0401] The memory 3001 may be a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 3001 may store a program, and when the program stored in the memory 3001 is executed by the processor 3002, the processor 3002 is used to execute the various steps of the method for constructing a neural network according to the embodiments of this application.

[0402] The processor 3002 may be a general-purpose central processing unit (CPU), microprocessor, application-specific integrated circuit (ASIC), graphics processing unit (GPU), or one or more integrated circuits, used to execute related programs to implement the method of constructing a neural network according to the method embodiments of this application.

[0403] The processor 3002 can also be an integrated circuit chip with signal processing capabilities; for example, it could be... Figure 5 The chip shown is an example. In implementation, each step of the method for constructing a neural network according to this application can be completed through integrated logic circuits in the hardware of the processor 3002 or through software instructions.

[0404] The processor 3002 described above can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules can be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. This storage medium is located in memory 3001. The processor 3002 reads the information in memory 3001 and, in conjunction with its hardware, completes the functions required by the units included in the device for constructing a neural network, or executes the method for constructing a neural network in the embodiments of this application.

[0405] The communication interface 3003 uses transceiver devices, such as, but not limited to, transceivers, to enable communication between the device 3000 and other devices or communication networks. For example, information about the neural network to be built and the training data needed in the process of building the neural network can be obtained through the communication interface 3003.

[0406] Bus 3004 may include a pathway for transmitting information between various components of device 3000 (e.g., memory 3001, processor 3002, communication interface 3003).

[0407] Figure 13 This is a schematic diagram of the hardware structure of the image processing apparatus according to an embodiment of this application. Figure 13 The image processing device 4000 shown includes a memory 4001, a processor 4002, a communication interface 4003, and a bus 4004. The memory 4001, processor 4002, and communication interface 4003 are interconnected via the bus 4004.

[0408] The memory 4001 can be a ROM, a static storage device, or RAM. The memory 4001 can store a program, and when the program stored in the memory 4001 is executed by the processor 4002, the processor 4002 and the communication interface 4003 are used to execute the various steps of the image processing method of the embodiments of this application.

[0409] The processor 4002 may be a general-purpose CPU, microprocessor, ASIC, GPU, or one or more integrated circuits, used to execute relevant programs to achieve the functions required by the units in the image processing apparatus of this application embodiment, or to execute the image processing method of this application method embodiment.

[0410] The processor 4002 can also be an integrated circuit chip with signal processing capabilities; for example, it could be... Figure 5 The chip shown. In implementation, each step of the image processing method of this application embodiment can be completed by the integrated logic circuit of the hardware in the processor 4002 or by instructions in the form of software.

[0411] The processor 4002 described above can also be a general-purpose processor, DSP, ASIC, FPGA, or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules can be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. This storage medium is located in memory 4001. The processor 4002 reads the information in memory 4001 and, in conjunction with its hardware, completes the functions required by the units included in the image processing apparatus of this application embodiment, or executes the image processing method of the method embodiment of this application.

[0412] The communication interface 4003 uses a transceiver device, such as, but not limited to, a transceiver, to enable communication between the device 4000 and other devices or communication networks. For example, an image to be processed can be acquired through the communication interface 4003.

[0413] Bus 4004 may include a pathway for transmitting information between various components of device 4000 (e.g., memory 4001, processor 4002, communication interface 4003).

[0414] Figure 14 This is a schematic diagram of the hardware structure of a neural network training device according to an embodiment of this application. Similar to devices 3000 and 4000 described above, Figure 14 The neural network training device 5000 shown includes a memory 5001, a processor 5002, a communication interface 5003, and a bus 5004. The memory 5001, processor 5002, and communication interface 5003 are interconnected via the bus 5004.

[0415] In passing Figure 12 The apparatus shown for constructing a neural network, once the neural network is constructed, can be used to... Figure 14 The neural network training device 5000 shown trains the neural network, and the trained neural network can then be used to execute the image processing method of the embodiments of this application.

[0416] Specifically, Figure 14 The device shown can acquire training data and the neural network to be trained from the outside through the communication interface 5003, and then the processor trains the neural network to be trained according to the training data.

[0417] It should be noted that although the above-described devices 3000, 4000, and 5000 only show a memory, processor, and communication interface, those skilled in the art should understand that in specific implementations, devices 3000, 4000, and 5000 may also include other devices necessary for normal operation. Furthermore, depending on specific needs, those skilled in the art should understand that devices 3000, 4000, and 5000 may also include hardware devices for implementing other additional functions. Moreover, those skilled in the art should understand that devices 3000, 4000, and 5000 may only include the devices necessary for implementing the embodiments of this application, and may not necessarily include... Figure 12 , Figure 13 and Figure 14 All the devices shown.

[0418] It should be understood that the processor in the embodiments of this application can be a central processing unit (CPU), but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0419] It should also be understood that the memory in the embodiments of this application can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDR SDRAM), enhanced synchronous DRAM (ESDRAM), synchronous linked DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0420] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.

[0421] It should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. A and B can be singular or plural. Additionally, the character " / " in this article generally indicates an "or" relationship between the preceding and following related objects, but it can also represent an "and / or" relationship. Please refer to the context for a more accurate understanding.

[0422] In this application, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or multiple items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0423] It should be understood that in the various embodiments of this application, the sequence number of each process does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0424] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0425] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0426] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0427] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0428] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0429] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0430] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An image processing method, characterized in that, include: Obtain the image to be processed; The image to be processed is classified according to the target neural network to obtain the classification result of the image to be processed; The target neural network is constructed from a target network structure, which is determined in the following way: Based on the application requirements of the target neural network, a search space is constructed, which includes M elements. The M elements are used to indicate M network structures. Each of the M elements includes the number of blocks in a stage of the corresponding network structure and the number of channels in the block. M is a positive integer. The target neural network is used for image classification. The application requirements of the target neural network include the running speed of the target neural network, the number of parameters of the target neural network, or the structural requirements of the target neural network. The structural requirements include the number of blocks in each stage of the structure of the target neural network and the number of channels in each block. The target network structure is selected from the M network structures based on the distribution relationship of the unevaluated elements in the search space, wherein the distribution relationship of the unevaluated elements is the clustering result of the unevaluated elements.

2. The method according to claim 1, characterized in that, The step of constructing the search space based on the application requirements of the target neural network includes: Based on the application requirements of the target neural network, an initial search space is constructed, which includes N initial elements. The N initial elements are used to indicate N initial network structures. Each of the N initial elements includes the number of blocks in the stage of the corresponding initial network structure and the number of channels in the block. N is a positive integer greater than or equal to M. The N initial network structures indicated by the N initial elements are filtered according to preset rules to obtain the M elements in the search space. The preset rules include: If the number of blocks in each stage of the first initial network structure indicated by the first initial element among the N initial elements is not greater than the number of blocks in the corresponding stage of the second initial network structure indicated by the second initial element among the N initial elements, and the number of channels in each block of each stage of the first initial network structure is not greater than the number of channels in each block of the corresponding stage of the second initial network structure, then the first initial element is deleted from the initial search space.

3. The method according to claim 1 or 2, characterized in that, The step of selecting the target network structure from the M network structures based on the distribution relationship of unevaluated elements in the search space includes: Based on the distribution relationship of the unevaluated elements, determine S elements among the unevaluated elements, where S is a positive integer less than M; The target network structure is selected from the M network structures based on the S elements.

4. The method according to claim 3, characterized in that, The step of selecting the target network structure from the M network structures based on the S elements includes: The S network structures indicated by the S elements in the unevaluated elements are evaluated to obtain the evaluation results of the evaluated elements, and the evaluation results of the evaluated elements include the evaluation results of the S network structures. Based on the evaluation results of the evaluated elements, the target network structure is selected from the M network structures.

5. The method according to claim 4, characterized in that, The step of selecting the target network structure from the M network structures based on the evaluation results of the evaluated elements includes: The first unevaluated element is modeled based on the evaluation results of the evaluated elements to obtain the model of the first unevaluated element. The first unevaluated element includes other elements in the search space besides the evaluated elements. Based on the model of the first unevaluated element, the target network structure is selected from the M network structures.

6. The method according to claim 5, characterized in that, If the target network structure cannot be selected from the M network structures based on the distribution of unevaluated elements in the search space, the target network structure is determined in the following way: Based on the distribution relationship of the first unevaluated element and the model of the first unevaluated element constructed based on the evaluated element, the target network structure is selected from the M network structures.

7. The method according to claim 6, characterized in that, The step of selecting the target network structure from the M network structures based on the distribution relationship of the first unevaluated elements and the model of the first unevaluated elements constructed based on the evaluated elements includes: Based on the distribution relationship of the first unevaluated elements, determine L elements among the first unevaluated elements, where L is a positive integer less than M; The target network structure is selected from the M network structures based on the L elements and the model of the first unevaluated elements constructed based on the evaluated elements.

8. The method according to claim 7, characterized in that, The step of selecting the target network structure from the M network structures based on the L elements and a model constructed based on the evaluated elements includes: Based on the model constructed from the first unevaluated element based on the evaluated element, determine Q elements from the L elements, where Q is a positive integer less than L; The Q network structures indicated by the Q elements are evaluated to obtain the evaluation result of the first evaluated element. The evaluation result of the first evaluated element includes the evaluation results of the S network structures and the evaluation results of the Q network structures. Based on the distribution relationship between the evaluation results of the first evaluated element and the second unevaluated element, the target network structure is selected from the M network structures. The second unevaluated element includes other elements in the search space besides the first evaluated element.

9. The method according to claim 7, characterized in that, The distribution relationship of the first unevaluated element is the clustering result of the first unevaluated element, and the L elements are elements in the L clusters included in the clustering result of the first unevaluated element.

10. The method according to claim 9, characterized in that, The L elements are the L elements corresponding to the centers of the L clusters.

11. The method according to claim 8, characterized in that, If the target network structure cannot be selected from the M network structures based on the distribution relationship of the first unevaluated elements and the model of the first unevaluated elements constructed based on the evaluated elements, the target network structure is determined in the following way: Based on the distribution relationship of the second unevaluated element and the model of the second unevaluated element constructed based on the first evaluated element, the target network structure is reselected from the M network structures.

12. The method according to any one of claims 6 to 10, characterized in that, The model based on the evaluated elements and the first unevaluated element includes: The model of the first unevaluated element is obtained based on the evaluation results of the evaluated network structure and the following formula: and / or ; in, This indicates the network structures in the search space that have not been evaluated. Indicates the precision threshold. This indicates that the precision is greater than the precision threshold. network structure The conditional probability density function, , This indicates that the precision is less than or equal to the precision threshold. network structure The conditional probability density function, , Indicates a mixture Gaussian distribution. , express and The Euclidean distance between them Indicates by The distance function formed , This indicates the first in the evaluated network structure. A network structure, This indicates the first in the evaluated network structure. The accuracy of the network structure express The corresponding weights Represents the normalization factor. express One of the hyperparameters, It is a positive integer. Let be the base of the natural logarithm function. Positively correlated with network structure performance It is positively correlated with the uncertainty of network structure performance.

13. The method according to any one of claims 6 to 10, characterized in that, The model based on the evaluated elements and the first unevaluated element includes: The model of the first unevaluated element is obtained based on the evaluation results of the evaluated network structure and the following formula: in, This represents the unevaluated elements in the search space; , Indicates the first of the evaluated elements One element, Indicates the first of the evaluated elements Precision of each element; , Represent the expected function; Let be a random variable that follows a Gaussian distribution. mean and variance and Input The following relationship must be satisfied: in, The number of network structures that have been evaluated. for The vector formed , , for The vector formed , , for The resulting matrix is, , , , As a hyperparameter, Indicates the first of the evaluated elements One element, Indicates and Same or different evaluated elements, is the base of the natural logarithm function; It is the identity matrix. Also a hyperparameter, It is a positive integer.

14. The method according to claim 3, characterized in that, The S elements are elements from the S clusters included in the clustering results of the unevaluated elements.

15. The method according to claim 14, characterized in that, The S elements are the S elements corresponding to the centers of the S clusters.

16. An image processing apparatus, characterized in that, include: The acquisition unit is used to acquire the image to be processed; An image processing unit is used to classify the image to be processed according to a target neural network to obtain a classification result of the image to be processed. The target neural network is constructed from a target network structure, which is determined in the following way: Based on the application requirements of the target neural network, a search space is constructed, which includes M elements. The M elements are used to indicate M network structures. Each of the M elements includes the number of blocks in a stage of the corresponding network structure and the number of channels in the block. M is a positive integer. The target neural network is used for image classification. The application requirements of the target neural network include the running speed of the target neural network, the number of parameters of the target neural network, or the structural requirements of the target neural network. The structural requirements include the number of blocks in each stage of the structure of the target neural network and the number of channels in each block. The target network structure is selected from the M network structures based on the distribution relationship of the unevaluated elements in the search space, wherein the distribution relationship of the unevaluated elements is the clustering result of the unevaluated elements.

17. The apparatus according to claim 16, characterized in that, The step of constructing the search space based on the application requirements of the target neural network includes: Based on the application requirements of the target neural network, an initial search space is constructed, which includes N initial elements. The N initial elements are used to indicate N initial network structures. Each of the N initial elements includes the number of blocks in the stage of the corresponding initial network structure and the number of channels in the block. N is a positive integer greater than or equal to M. The N initial network structures indicated by the N initial elements are filtered according to preset rules to obtain the M elements in the search space. The preset rules include: If the number of blocks in each stage of the first initial network structure indicated by the first initial element among the N initial elements is not greater than the number of blocks in the corresponding stage of the second initial network structure indicated by the second initial element among the N initial elements, and the number of channels in each block of each stage of the first initial network structure is not greater than the number of channels in each block of the corresponding stage of the second initial network structure, then the first initial element is deleted from the initial search space.

18. The apparatus according to claim 16 or 17, characterized in that, The step of selecting the target network structure from the M network structures based on the distribution relationship of unevaluated elements in the search space includes: Based on the distribution relationship of the unevaluated elements, determine S elements among the unevaluated elements, where S is a positive integer less than M; The target network structure is selected from the M network structures based on the S elements.

19. The apparatus according to claim 18, characterized in that, The step of selecting the target network structure from the M network structures based on the S elements includes: The S network structures indicated by the S elements in the unevaluated elements are evaluated to obtain the evaluation results of the evaluated elements, and the evaluation results of the evaluated elements include the evaluation results of the S network structures. Based on the evaluation results of the evaluated elements, the target network structure is selected from the M network structures.

20. The apparatus according to claim 19, characterized in that, The step of selecting the target network structure from the M network structures based on the evaluation results of the evaluated elements includes: The first unevaluated element is modeled based on the evaluation results of the evaluated elements to obtain the model of the first unevaluated element. The first unevaluated element includes other elements in the search space besides the evaluated elements. Based on the model of the first unevaluated element, the target network structure is selected from the M network structures.

21. The apparatus according to claim 20, characterized in that, If the target network structure cannot be selected from the M network structures based on the distribution of unevaluated elements in the search space, the target network structure is determined in the following way: Based on the distribution relationship of the first unevaluated element and the model of the first unevaluated element constructed based on the evaluated element, the target network structure is selected from the M network structures.

22. The apparatus according to claim 21, characterized in that, The step of selecting the target network structure from the M network structures based on the distribution relationship of the first unevaluated elements and the model of the first unevaluated elements constructed based on the evaluated elements includes: Based on the distribution relationship of the first unevaluated elements, determine L elements among the first unevaluated elements, where L is a positive integer less than M; The target network structure is selected from the M network structures based on the L elements and the model of the first unevaluated elements constructed based on the evaluated elements.

23. The apparatus according to claim 22, characterized in that, The step of selecting the target network structure from the M network structures based on the L elements and a model constructed based on the evaluated elements includes: Based on the model constructed from the first unevaluated element based on the evaluated element, determine Q elements from the L elements, where Q is a positive integer less than L; The Q network structures indicated by the Q elements are evaluated to obtain the evaluation result of the first evaluated element. The evaluation result of the first evaluated element includes the evaluation results of the S network structures and the evaluation results of the Q network structures. Based on the distribution relationship between the evaluation results of the first evaluated element and the second unevaluated element, the target network structure is selected from the M network structures. The second unevaluated element includes other elements in the search space besides the first evaluated element.

24. The apparatus according to claim 22, characterized in that, The distribution relationship of the first unevaluated element is the clustering result of the first unevaluated element, and the L elements are elements in the L clusters included in the clustering result of the first unevaluated element.

25. The apparatus according to claim 24, characterized in that, The L elements are the L elements corresponding to the centers of the L clusters.

26. The apparatus according to claim 23, characterized in that, If the target network structure cannot be selected from the M network structures based on the distribution relationship of the first unevaluated elements and the model of the first unevaluated elements constructed based on the evaluated elements, the target network structure is determined in the following way: Based on the distribution relationship of the second unevaluated element and the model of the second unevaluated element constructed based on the first evaluated element, the target network structure is reselected from the M network structures.

27. The apparatus according to any one of claims 20 to 26, characterized in that, The model, built based on the evaluated elements and consisting of the first unevaluated element, includes: The model of the first unevaluated element is obtained based on the evaluation results of the evaluated network structure and the following formula: and / or ; in, This indicates the network structures in the search space that have not been evaluated. Indicates the precision threshold. This indicates that the precision is greater than the precision threshold. network structure The conditional probability density function, , This indicates that the precision is less than or equal to the precision threshold. network structure The conditional probability density function, , Indicates a mixture Gaussian distribution. , express and The Euclidean distance between them Indicates by The distance function formed , This indicates the first in the evaluated network structure. A network structure, This indicates the first in the evaluated network structure. The accuracy of the network structure express The corresponding weights Represents the normalization factor. express A hyperparameter, It is a positive integer. Let be the base of the natural logarithm function. Positively correlated with network structure performance It is positively correlated with the uncertainty of network structure performance.

28. The apparatus according to any one of claims 20 to 26, characterized in that, The model, built based on the evaluated elements and consisting of the first unevaluated element, includes: The model of the first unevaluated element is obtained based on the evaluation results of the evaluated network structure and the following formula: in, This represents the unevaluated elements in the search space; , Indicates the first of the evaluated elements One element, Indicates the first of the evaluated elements Precision of each element; , Represent the expected function; Let be a random variable that follows a Gaussian distribution. mean and variance and Input The following relationship must be satisfied: in, The number of network structures that have been evaluated. for The vector formed , , for The vector formed , , for The resulting matrix is, , , , As a hyperparameter, Indicates the first of the evaluated elements One element, Indicates and Same or different evaluated elements, is the base of the natural logarithm function; It is the identity matrix. Also a hyperparameter, It is a positive integer.

29. The apparatus according to claim 18, characterized in that, The S elements are elements from the S clusters included in the clustering results of the unevaluated elements.

30. The apparatus according to claim 29, characterized in that, The S elements are the S elements corresponding to the centers of the S clusters.

31. An image enhancement device, characterized in that, It includes a processor and a memory, the memory being used to store program instructions, and the processor being used to invoke the program instructions to perform the method of any one of claims 1 to 15.

32. A computer-readable storage medium, characterized in that, The computer-readable medium stores program code for execution by the device, the program code including methods for performing any one of claims 1 to 15.

33. A chip, characterized in that, The chip includes a processor and a data interface, wherein the processor reads instructions stored in the memory through the data interface to execute the method as described in any one of claims 1 to 15.

Images