A model searching method, device, apparatus and storage medium

Abstract

This application discloses a method, apparatus, device, and storage medium for model search. The method includes: acquiring a search space, which includes a convolutional module and a visual transformation module; wherein the visual transformation module includes a self-attention sub-module for executing a self-attention algorithm with a complexity of O(n); preprocessing labeled training images using the convolutional module; performing target task-based image processing on the preprocessed results using the visual transformation module; determining a first gradient of the search space based on a first loss between the image processing results and the labels of the training images; updating the search space based on the first gradient to obtain a target search space; wherein the target search space is used to search for a target model, and the target model is used to perform target task-based image processing on the target image to be processed. This allows the searched model to have linear complexity, effectively solving the problem of difficult model deployment.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method, apparatus, device, and storage medium for model search. Background Technology

[0002] Neural network models based on visual transformers (hereinafter referred to as Transformer models) can be used for a variety of image processing tasks, such as image classification, image detection, and image segmentation.

[0003] To find the optimal network structure for the Transformer model, Neural Architecture Search (NAS) can be used to automatically tune the Transformer model by constructing a search space. However, existing NAS methods for Transformer models involve high-complexity search spaces, typically O(n^2). 2 Therefore, it requires a huge amount of computing power and storage, making it difficult to deploy on the edge. Summary of the Invention

[0004] This application provides at least one method, apparatus, device, and storage medium for model search.

[0005] The technical solution of this application embodiment is implemented as follows:

[0006] In a first aspect, embodiments of this application provide a method for model search, the method comprising: obtaining a search space, the search space including a convolutional module and a visual transformation module; wherein the visual transformation module includes a self-attention sub-module for performing a self-attention algorithm, the self-attention algorithm having a complexity of O(n); preprocessing labeled training images using the convolutional module; performing target task-based image processing on the preprocessed result using the visual transformation module; determining a first gradient of the search space based on a first loss between the image processing result and the labeled training images; updating the search space based on the first gradient to obtain a target search space; wherein the target search space is used to search for a target model, and the target model is used to perform target task-based image processing on the target image to be processed.

[0007] Secondly, embodiments of this application provide a model search apparatus, comprising: an acquisition module for acquiring a search space, the search space including a convolution module and a visual transformation module; wherein the visual transformation module includes a self-attention sub-module for executing a self-attention algorithm, the self-attention algorithm having a complexity of O(n); a preprocessing module for preprocessing labeled training images using the convolution module; an image processing module for performing target task-based image processing on the preprocessed results using the visual transformation module; a first determination module for determining a first gradient of the search space based on a first loss between the image processing results and the labels of the training images; and a gradient update module for updating the search space based on the first gradient to obtain a target search space; wherein the target search space is used to search for a target model, and the target model is used to perform target task-based image processing on the target image to be processed.

[0008] Thirdly, embodiments of this application provide a model search device, which includes a memory and a processor; wherein the memory is used to store computer-executable instructions; and the processor is connected to the memory and is used to implement the method described in the first aspect by executing the computer-executable instructions.

[0009] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by at least one processor, implements the method described in the first aspect.

[0010] This application provides a method, apparatus, device, and computer-readable storage medium for model search. It acquires a search space, which includes a convolutional module and a visual transformation module. The visual transformation module includes a self-attention submodule for executing a self-attention algorithm with a complexity of O(n). During training the search space, the convolutional module preprocesses the labeled training images. The visual transformation module performs target task-based image processing on the preprocessed results. A first gradient of the search space is determined based on a first loss between the image processing results and the labeled training images. The search space is updated based on the first gradient to obtain a target search space. This target search space is used to search for a target model, and the target model is used to perform target task-based image processing on the target image to be processed. This yields a search space with linear complexity, enabling the final searched target model to have linear complexity. This reduces the computational and storage requirements of the deployment device and effectively solves the problem of difficulty in deploying the searched target model on the edge. Attached Figure Description

[0011] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with this application and, together with the specification, serve to explain the technical solutions of this application.

[0012] Figure 1 A flowchart illustrating a model search method provided in an embodiment of this application;

[0013] Figure 2 This is a schematic diagram illustrating an example of the processing procedure of the self-attention submodule in an embodiment of this application;

[0014] Figure 3 A schematic diagram of a possible implementation flow of the model selection method provided in the embodiments of this application;

[0015] Figure 4 This is a schematic diagram of an example of the search space in an embodiment of this application;

[0016] Figure 5 This is a flowchart illustrating the gradient consistency distillation training method according to an embodiment of this application;

[0017] Figure 6 This is a schematic diagram illustrating an example of gradient backpropagation in the teacher and student models of this application embodiment;

[0018] Figure 7 A schematic diagram of the composition structure of a model search device provided in an embodiment of this application;

[0019] Figure 8 This is a schematic diagram of a hardware entity of the model search device in an embodiment of this application. Detailed Implementation

[0020] In order to gain a more detailed understanding of the features and technical content of the embodiments of this application, the implementation of the embodiments of this application will be described in detail below with reference to the accompanying drawings. The accompanying drawings are for reference and illustration only and are not intended to limit the embodiments of this application.

[0021] Unless otherwise defined, all technical and scientific terms used in the embodiments of this application have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in the embodiments of this application is for the purpose of describing the embodiments of this application only and is not intended to limit this application.

[0022] In the following description, references to "some embodiments" refer to a subset of all possible embodiments. It is understood that "some embodiments" may be the same or different subsets of all possible embodiments and may be combined with each other without conflict. It should also be noted that the terms "first, second, third" used in the embodiments of this application are merely for distinguishing similar objects and do not represent a specific ordering of objects. It is understood that "first, second, third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0023] It should be understood that the term "and / or" in the embodiments of this application is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. In addition, the character " / " in this document generally indicates that the related objects before and after it have an "or" relationship.

[0024] The Transformer model can be used for various image processing tasks, such as image classification, image detection, and image segmentation. However, the network structure of the Transformer model is hand-designed, which limits the improvement of model performance due to the fixed, hand-designed network based on prior knowledge. To find the optimal network structure for the Transformer model, a network structure search method can be used to automatically tune the Transformer model by constructing a search space. For example, one search method can train multiple networks at once, sampling subnets from the supernet for training each time. All subnets inherit the weights of the corresponding parts of the supernet, and then perform an evolutionary search under resource constraints to obtain the desired network.

[0025] Existing network architecture search methods for Transformer models have high search space complexity, typically O(n^2). 2 Therefore, it requires enormous computing power and storage, making edge deployment difficult. Limiting the number of parameters and computational cost of this method to an acceptable range will result in poor accuracy of the searched model. The high search space complexity in existing methods stems from the inclusion of standard self-attention mechanisms. For clarity, a brief introduction to standard self-attention mechanisms is provided below.

[0026] The parameters included in the standard self-attention mechanism are shown in Equation (1):

[0027]

[0028] Where, X∈R n×dThe input consists of a sequence of n tokens, each token being a d-dimensional feature vector. W q W k W v ∈R d×dm There are three learnable embedding matrices, d m The embedding dimension is represented by equation (2). The self-attention mechanism can be represented by equation (2):

[0029]

[0030] Standard self-attention mechanisms, because they directly use matrix dot product operations, have a computational and memory complexity of O(n^2). 2 This makes it difficult to deploy on edge devices. If this technology is used to implement a lightweight visual Transformer model (VisionTransformer, ViT), extensive downsampling is required upfront, resulting in significant loss of image information and reduced model accuracy. In other words, forcibly limiting the number of parameters and computational cost of the standard self-attention mechanism to an acceptable range leads to poor model accuracy.

[0031] To solve this problem, a Gaussian kernel function can be used instead of the matrix dot product, allowing the self-attention matrix to be approximated by a low-rank decomposition. The self-attention mechanism can then be expressed by equation (3):

[0032] Attention(Q,K,V)=exp(QθK)V(3);

[0033] Where S = exp(QθK), exp(·θ·) is the Gaussian kernel function, and exp(QθK) represents the operation performed on Q and K using the Gaussian kernel function. S∈R n×n The elements in the matrix can be represented as:

[0034]

[0035] Among them, Q i,: K represents the element in the i-th row of matrix Q. :,j Let represent the j-th column element of matrix K. Then, according to... The method obtains samples from Q. And sampled from K Let Q = K, then we get:

[0036]

[0037] in, express The Moore-Penrose inverse matrix can be solved iteratively using the Newton–Raphson method. This is used to approximate S. Therefore, the self-attention mechanism can be expressed as:

[0038]

[0039] After approximation, by calculating the contents inside the square brackets before and after the approximation, the complexity of the self-attention mechanism can be reduced to O(n).

[0040] In view of this, embodiments of this application provide a method, apparatus, device, and storage medium for model search. In this method, a search space including a convolutional module and a visual transformation module can be obtained. The visual transformation module may include a self-attention submodule for executing a self-attention algorithm with a complexity of O(n). This yields a search space with linear complexity (i.e., O(n)), thereby enabling the finally searched target model to possess linear time and space complexity. This reduces the computational and storage requirements of the deployment device and effectively solves the problem of difficulty in deploying the searched target model on the edge (such as a mobile device).

[0041] The embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0042] Figure 1 This application illustrates a method for model search according to an embodiment of the present application. This method can be executed, for example, by a model search device, or by a chip, chip system, or circuit configured in the model search device; the present application does not limit this. The model search device can refer to a device with data processing capabilities, such as a server, laptop, tablet, desktop computer, or mobile phone.

[0043] like Figure 1 As shown, the method may include:

[0044] S101, Obtain the search space, which includes a convolution module and a visual transformation module; wherein, the visual transformation module includes a self-attention sub-module for executing the self-attention algorithm, and the complexity of the self-attention algorithm is O(n).

[0045] The convolution module can be used to preprocess the input image (such as through downsampling), and the visual transformation module can be used to perform image processing based on the target task on the preprocessed result of the convolution module. For example, the target task may include, but is not limited to, any of the following: image classification task, image detection task, or image segmentation task.

[0046] In one possible approach, a self-attention algorithm with a complexity of O(n) can be implemented, for example, using the method in equation (6). Let Q = K in equation (6), then we obtain equation (7):

[0047]

[0048] in, O indicates the output result.

[0049] Based on formula (7), the self-attention submodule can perform the following operations on the preprocessed result output by the convolution module: multiply the first weight matrix with the preprocessed result of the convolution module to obtain the first feature matrix; multiply the second weight matrix with the preprocessed result of the convolution module to obtain the second feature matrix; sample the first feature matrix to obtain the third feature matrix; perform the first operation on the first feature matrix and the third feature matrix based on the Gaussian kernel function to obtain the first operation result; perform the second operation on the first operation result and the second feature matrix to obtain the output result of the self-attention submodule. The following is combined with... Figure 2 The above operations are illustrated by example.

[0050] Figure 2 This is a schematic diagram illustrating an example of the processing procedure of the self-attention submodule in an embodiment of this application. As an example, the preprocessing result of the convolution module is denoted as X, and the first weight is denoted as W. q The second weight, for example, is denoted as W. v By using the first weight W q Multiplying by X yields the first characteristic matrix Q; the second weight matrix W... v Multiplying by X yields the second characteristic matrix V. Sampling the first characteristic matrix Q yields the third characteristic matrix. The sampling method can be, for example, the following methods. Method. Then, the first eigenma matrix Q and the third eigenma matrix can be analyzed based on the Gaussian kernel function exp(·θ·). Perform the first operation to obtain the first result. This first operation may include: applying the Gaussian kernel function exp(·θ·) to Q and... Processing is performed to obtain and Using the Newton-Raphson method based on Calculated The first calculation result can be obtained through the above steps: and Then, a second operation can be performed on the first operation result and the second feature matrix V to obtain the output result of the self-attention submodule. The second operation may include: [the following steps are not explicitly stated in the original text, but can be omitted:] ...from the first operation result... and Multiply, the result of which is denoted as Y; and combine the results of the first operation. Multiply the result of the first feature matrix V by the second feature matrix, and denote the result as Z. Multiply Y and Z to obtain the output result O of the self-attention submodule.

[0051] It should be understood that, in the embodiments of this application, in addition to using Gaussian kernels to implement a self-attention algorithm with linear complexity, other matrix factorization linearization methods can also be used to make the complexity of the self-attention algorithm O(n), and the embodiments of this application do not limit this.

[0052] According to the method of this embodiment, the visual conversion module may include a self-attention submodule for executing a self-attention algorithm with a complexity of O(n). In this way, a search space with linear complexity can be obtained, which in turn enables the final searched target model to have linear time and space complexity, thereby reducing the computing power and storage requirements of the deployment device and effectively solving the problem that the searched target model is difficult to deploy on the edge.

[0053] After obtaining the search space, it can be trained to obtain the target search space. The target search space can be used to search for the target model. The steps for training the search space may include S102 to S105:

[0054] S102 uses a convolution module to preprocess the labeled training images.

[0055] In embodiments of this application, corresponding training images can be determined based on the target task. For example, when the target task is an image classification task, the training images may include images of multiple categories, and each image may correspond to a specific category as its label. In other words, for an image classification task, the true category of the training image is its label.

[0056] In S102, a convolution module can be used to preprocess the labeled training images, such as the following sampling process.

[0057] Figure 4 Figure (b) illustrates one possible model structure for a convolutional module. The convolutional module can, for example, comprise M blocks, where M is a positive integer. As an example, each block can consist of three concatenated convolutional layers: a 1×1 convolutional layer, a K×K convolutional layer, and a 1×1 convolutional layer. Figure 4In Figure (b), when there is no downsampling operation, the dashed line represents the residual operation, and the convolutional module is a ResNet module; when there is a downsampling operation, the dashed line represents that no operation is performed. For example, the searchable model parameters (model structure parameters) in the convolutional module may include, but are not limited to, at least one of the following: kernel size, number of channels, channel scaling factor, and number of blocks.

[0058] It should be understood that the embodiments of this application do not limit the number of convolutional modules in the search space. For example, the search space may include one convolutional module, or it may include two or more convolutional modules, and the specific number can be determined according to the actual application scenario and application requirements.

[0059] According to the method of this embodiment, by using a convolution module to preprocess the image, the computational load can be saved by utilizing the inductive bias of the convolution operation on the image task.

[0060] S103 uses a visual transformation module to perform target task-based image processing on the preprocessing results of the convolution module.

[0061] After image preprocessing, a visual transformation module can be used to perform task-based image processing on the preprocessed results. For example, when the target task is image classification, the visual transformation module can classify the input training images to estimate the category of the training images (or the probability that the training images belong to a certain category), and use the estimated category (or probability) as the image processing result of the visual transformation module. It should be understood that the image processing result of the visual transformation module can also be understood as the image processing result of the search space.

[0062] As one implementation, the visual transformation module can be, for example, a Transformer model, and the complexity of the self-attention algorithm part in the Transformer model is O(n).

[0063] Figure 4 Figure (c) shows the visual conversion module (corresponding to...) Figure 4 This is one possible model structure for a linear Transformer module. As an example, each visual transformation module can include N blocks, where N is a positive integer. In one implementation, each block can consist of a linear self-attention layer (i.e., a self-attention sub-module), a feedforward layer, and two residual and normalization (Add & Norm) layers. The connections between these layers can be found in [reference needed]. Figure 4 The details of Figure (c) are omitted here.

[0064] For example, the searchable model parameters (model structure parameters) in the visual transformation module include, but are not limited to, at least one of the following: embedding dimension, feed forward ratio, and number of blocks. The number of convolutional modules, the number of blocks in each convolutional module (e.g., denoted as M), the number of visual transformation modules, and the number of blocks in each visual transformation module (e.g., denoted as N) collectively determine the model depth. As an example, the model depth can be expressed as: Model depth = Number of convolutional modules × M + Number of visual transformation modules × N. In some embodiments, when the self-attention algorithm executed by the self-attention submodule is a multi-head self-attention algorithm, the searchable model parameters may further include the number of heads (HeadNum).

[0065] It should be understood that the embodiments of this application do not limit the number of visual transformation modules in the search space. For example, the search space may include one visual transformation module, or it may include two or more visual transformation modules, and the specific number can be determined according to the actual application scenario and application requirements.

[0066] S104, the first gradient of the search space is determined based on the first loss between the image processing results of the visual transformation module and the annotations of the training images.

[0067] One implementation approach involves first calculating the first loss between the image processing results of the visual conversion module and the annotations of the training images, using the corresponding loss function. Further, by differentiating the weights of the search space with respect to the first loss, the first gradient of the search space can be obtained.

[0068] S105, update the search space based on the first gradient to obtain the target search space.

[0069] In this step, the search space can be updated based on the first gradient calculated in S104 to obtain the target search space. The target search space can be used to search for a target model, and the target model can be used to perform target task-based image processing on the target image to be processed.

[0070] It should be noted that multiple rounds of gradient updates can be performed on the search space during training. For example, after completing one round of gradient updates, the resulting search space can be used as input for the next round of training, and a new round of gradient updates can be performed. The iteration can stop when the number of update rounds reaches a preset upper limit, or when the model convergence condition is met, and the final target search space can be output.

[0071] In some embodiments, the search space can serve as a teacher model, and at least one sub-model in the search space can serve as a student model. In this case, updating the search space based on a first gradient to obtain the target search space includes: updating the teacher model based on the first gradient, and updating the corresponding student model based on a second gradient of each student model to obtain the target search space.

[0072] In other words, when training the search space, at least one sub-model can be sampled from the search space, which can serve as a student model. Simultaneously, the search space can be used as a teacher model. Therefore, the search space can be trained by simultaneously updating the gradients of both the teacher and student models, thereby updating the entire search space. In this case, the method may further include steps 51) to 53).

[0073] 51) Determine at least one sub-model in the search space as a student model; wherein each student model includes a convolutional sub-module obtained by sampling the convolutional module and a visual transformation sub-module obtained by sampling the visual transformation module.

[0074] In one possible approach, at least one sub-model in the search space can be identified as a student model through the following steps: a) obtaining a candidate parameter set, which includes model parameters that can be used by the convolutional sub-module and the visual transformation sub-module; b) determining the target model parameters selected by the convolutional sub-module and the target model parameters selected by the visual transformation sub-module from the candidate parameter set; c) sampling the search space based on the target model parameters selected by the convolutional sub-module and the target model parameters selected by the visual transformation sub-module to obtain the corresponding sub-model, and identifying the sub-model as the student model.

[0075] For example, in step a) above, the model parameters that the convolution module can use may include, but are not limited to, at least one of the following: kernel size, number of channels, channel scaling ratio, and number of blocks; the model parameters that the visual conversion module can use may include, but are not limited to, at least one of the following: embedding dimension, feedforward ratio, model depth, and number of heads used in the self-attention algorithm. The selectable numerical range for each model parameter can be set based on the actual application scenario and application requirements.

[0076] In step b) above, the target model parameters for the convolutional submodule and the visual transformation submodule can be determined from the candidate parameter set. For example, the target model parameters for the convolutional submodule might include: kernel size = 5, number of channels = 128, channel scaling ratio = 0.8, and number of blocks = 3; the target model parameters for the visual transformation submodule might include: embedding dimension = 448, feedforward ratio = 3.5, number of blocks = 4, and number of heads used in the self-attention algorithm = 8.

[0077] In step c) above, the corresponding sub-model can be sampled from the search space according to the target model parameters selected for the convolutional sub-module and the visual transformation sub-module determined in step b), and this sub-model is determined as the student model. The model parameters of the sampled sub-model include the target model parameters; in other words, the values ​​of the sampled sub-model's model parameters are the values ​​of the selected target model parameters.

[0078] 52) Distill each student model using the teacher model, and determine the second gradient of each student model based on the distillation results (e.g., denoted as G). stu ).

[0079] In one possible approach, the teacher model is used to distill each student model, including: for each student model, determining the third gradient of the student model (e.g., denoted as G) based on a second loss between the image processing results of the student model and the image processing results of the teacher model. distill The method for calculating the second loss can be, for example, calculating the difference between the image processing results of the student model and the teacher model. The method for calculating the third gradient can be, for example, taking the derivative of the second loss with respect to the weights of the student model.

[0080] In one possible approach, the step of determining the second gradient of each student model based on the results of distillation includes: d) for each student model, determining the fourth gradient of the student model (e.g., denoted as G) based on the third loss between the image processing results of the student model and the annotations of the training images. s-label e) in the first gradient (e.g., denoted as G) t-label If the first gradient and the fourth gradient are in the same direction, the sum of the third and fourth gradients is determined as the second gradient of the student model; if the first gradient and the fourth gradient are in different directions, the third gradient is determined as the second gradient of the student model. The first gradient is determined based on the first loss between the search space (teacher model) and the labels of the training images (or the true values / standard values). Therefore, the first gradient can also be understood as the gradient of the search space (teacher model) relative to the labels of the training images.

[0081] In step d) above, the method for determining the fourth gradient based on the third loss can be, for example, by calculating the third loss between the image processing results of the student model and the annotations of the training images, and combining the corresponding loss function; further, by taking the derivative of the third loss with respect to the weights of the student model, the fourth gradient of the student model can be obtained.

[0082] In step e) above, the first gradient G can be used as a reference. t-label With the fourth gradient G s-label Whether the directions are consistent, determine the second gradient G used for gradient update of the student model. stu .

[0083] One possible scenario is the first gradient G. t-label and the fourth gradient G s-label The directions are consistent. In this case, the third gradient G can be... distill With the fourth gradient G s-label The summation result is determined as the second gradient G of the student model. stu That is, G stu =G distill +G s-label .

[0084] Another possible scenario is the first gradient G. t-label and the fourth gradient G s-label The directions are inconsistent. In this case, the third gradient G can be... distill The second gradient G of the student model was determined. stu That is, G stu =G distill .

[0085] In some embodiments, the method further includes: for each fourth gradient, determining the cosine value between the first gradient and the fourth gradient. In this case, the direction of the first gradient and the fourth gradient can be determined by the following rule: if the cosine value between the first gradient and the fourth gradient is greater than or equal to zero, the direction of the first gradient and the fourth gradient is determined to be consistent; if the cosine value between the first gradient and the fourth gradient is less than zero, the direction of the first gradient and the fourth gradient is determined to be inconsistent. That is, if cos(G t-label G s-label If )≥0, then G can be determined. t-label With G s-label If the directions are the same, and cos(G) t-label G s-label If ) < 0, then G can be determined. t-label With G s-label The directions are inconsistent.

[0086] 53) Update the teacher model based on the first gradient, and update the corresponding student model based on the second gradient of each student model to obtain the target search space.

[0087] It should be noted that gradient updates for both the teacher and student models can be performed in multiple rounds, or in other words, it can be iterated multiple times. The iteration can stop when the preset maximum number of rounds is reached, or when the model convergence condition is met, and the final target search space can be output.

[0088] According to the method of this embodiment, the teacher model performs distillation on each student model, and the student model is updated with gradients based on the results of the distillation. This ensures that the gradient updates of the teacher model and each student model are performed in the same direction, thus ensuring that the gradient updates of the entire search space are performed in the same direction, thereby avoiding the problem of mutual interference between gradients of the models.

[0089] In some embodiments, the method further includes S106:

[0090] S106, based on the search algorithm, searches for the target model from the target search space.

[0091] For example, the search algorithm may include, but is not limited to, any of the following: evolutionary search algorithm (or evolutionary algorithm), random search algorithm, and reinforcement learning search algorithm. The following description uses a search algorithm as an example.

[0092] In one possible approach, the steps of using an evolutionary search algorithm to search for the target model from the target search space may include steps 61) to 63):

[0093] 61) Sample the target search space to obtain at least two seed models, and perform recombination and / or mutation operations on each seed model to obtain at least two offspring models.

[0094] The recombination operation may include: exchanging at least one pair of model parameters in the model; the mutation operation may include: randomly changing at least one model parameter within a predefined numerical range. Here, model parameters can also be understood as the values ​​that model parameters take.

[0095] In some embodiments, sampling the target search space to obtain at least two seed models includes: sampling the target search space based on model constraints to obtain at least two seed models that satisfy the model constraints; wherein, the model constraints may include, for example, at least one of the following: the number of model parameters is less than or equal to a preset number of parameters; when running the model, the hardware latency is less than or equal to a preset latency; when running the model, the number of floating-point operations per second (FLOPs) performed by the hardware is less than or equal to a preset number.

[0096] According to the method of this embodiment, during the sampling of seed models, by introducing information such as model parameter quantity, FLOPs, and hardware latency, the lightweight nature of the model can be controlled, thereby determining the target model that meets the corresponding deployment conditions (constraints) for deployment devices with different hardware resources. In other words, the target model that meets the corresponding deployment conditions in this embodiment can be deployed in the corresponding deployment device. The deployment conditions may include, for example, the hardware resources of the deployment device and the network accuracy and / or computational load conditions set by the business scenario.

[0097] 62) Using a pre-trained validation model, estimate the accuracy of image processing results for at least two child models.

[0098] In evolutionary search algorithms, to evaluate the performance of the sampled seed networks, a pre-trained validation model can be used to estimate the accuracy of the image processing results for each seed network. In one possible approach, training the validation model may include: f) randomly selecting T sub-models from the target search space and validating them using a labeled validation dataset to obtain the accuracy of the image processing results for each of the T sub-models; g) combining the model parameters of the T sub-models with their accuracy rates to form a training set; h) constructing a validation model and training it using the training set to obtain a validation model that can predict the model accuracy based on the model parameters.

[0099] 63) Determine the target model from at least two child models based on the accuracy of the image processing results of at least two child models.

[0100] As an example, the model with the higher accuracy in image processing results among at least two child models can be identified as the target model.

[0101] It should be understood that steps 61) to 63) above can be executed in multiple rounds, or in other words, it can be iterated multiple times, thus yielding multiple target models. In this case, the model with the highest image processing accuracy can be selected from these multiple target models as the final target model.

[0102] The foregoing described a model selection method provided by an embodiment of this application. To facilitate understanding of the embodiments of this application, the following will be combined with... Figures 3 to 6 This paper introduces a possible implementation process of the model selection method provided in the embodiments of this application.

[0103] Figure 3 This is a schematic diagram illustrating a possible implementation flow of the model selection method provided in an embodiment of this application. Figure 3 As shown, the implementation process may include:

[0104] S301, Obtain search space.

[0105] In this step, a search space with linear complexity can be obtained. The device executing the method of this embodiment can generate its own search space or receive a search space from other devices; there is no limitation on this.

[0106] Figure 4 This is a schematic diagram of an example of the search space in an embodiment of this application. The search space can be divided into two main parts, the first part being the convolution module (corresponding to...). Figure 4 The first part (S1) is used to perform convolution operations; the second part is the linear Transformer module (corresponding to...). Figure 4 The S2 / S3 / S4 modules are used for task-based image processing of the target image. This structural design is similar to the LeViT model. The first part uses convolutional modules to save computation by leveraging the inductive bias of convolution operations on the image task. Depending on the hardware characteristics, the convolutional modules can be ResNet, MBConv, or other types of convolutional modules; there are no restrictions. The second part uses a linear Transformer, divided into a linear multi-head self-attention mechanism (corresponding to...). Figure 4 The linear self-attention layer and the multi-layer perceptron (MLP) part (corresponding to) Figure 4 (The feedforward layer in the algorithm). The input to this search space can be, for example, labeled training images used to train the search space. The output of S1 can be used as the input feature map of S2.

[0107] like Figure 4 As shown, the search space may include at least one convolutional module, for example in Figure 4 The search space shown may include one convolutional module S1. The structure of S1 is as follows: Figure 4As shown in Figure (b), the convolutional module can include M blocks, that is, the convolutional module can be composed of M blocks. As one implementation, each block can consist of three concatenated convolutional layers: a 1×1 convolutional layer, a K×K convolutional layer, and a 1×1 convolutional layer. In S1, when there is no downsampling operation, the dashed line represents the residual operation; in this case, S1 is a ResNet module. When there is a downsampling operation, the dashed line represents that no operation is performed.

[0108] For example, the parameters to be searched in S1 include: kernel size K = {3,5,7}, number of channels = {32,64,128,256}, channel scaling factor e = {0.4,0.6,0.8,1.0}, and number of blocks M.

[0109] The search space may also include at least one linear Transformer module, for example in Figure 3 The search space shown can include three linear Transformer modules, namely S2, S3, and S4. The structure of each linear Transformer module is as follows: Figure 4 As shown in Figure (c), each linear Transformer module can include N blocks; that is, each linear Transformer module can consist of N blocks. As one implementation, each block can consist of a linear self-attention layer, a feedforward layer, and two residual and normalization layers. The connection relationships between the layers can be found in [reference needed]. Figure 4 The details of Figure (c) are omitted here. Figure 4 The processing of the input feature map by the linear self-attention layer is as follows: Figure 2 As shown, arrows without special markings indicate matrix multiplication.

[0110] For example, the parameters to be searched in S2-S4 include: a. Embedding dimension = [192, 448]. The embedding dimension ranges from 192 to 448, and a step size of 32 can be used during the search. Based on the selected embedding dimension, the input feature map of S2 can be divided into n parts (n is the value of the embedding dimension), and the number of pixels in each part can be determined by the embedding dimension and the size of the feature map. b. Feedforward ratio = [3, 4]. During the search, the feedforward part can use a step size of 0.5 within the range of 3 to 4 to determine the scaling ratio (i.e., the feedforward ratio). c. Number of heads = [4, 8]. When the self-attention algorithm executed by the linear self-attention layer is a multi-head linear self-attention algorithm, the parameters to be searched also include the number of heads, which can be selected within the range of 4 to 8 using a step size of 1. d. Model depth. Model depth is determined by the number of convolutional modules, the number of blocks in each convolutional module (M), the number of linear Transformer modules, and the number of blocks in each linear Transformer module (N). As an example, model depth can be expressed as: Model Depth = Number of Convolutional Modules × M + Number of Linear Transformer Modules × N. For example, in this embodiment, the number of convolutional modules is 1, and the number of linear Transformer modules is 3, then the model depth = M + 3N. As an example, the searchable ranges for M and N are M = [3,5] and N = [3,5], respectively. During the search, the depth of each stage can be selected within the range of 3 to 5 with a stride of 1.

[0111] It should be understood that the parameters to be searched and the range of their values ​​in the embodiments of this application are merely exemplary and should not constitute any limitation on the implementation process of the embodiments of this application.

[0112] S302, train the search space to obtain the target search space.

[0113] After obtaining the search space, the entire search space needs to be trained for use in subsequent search processes. In this embodiment, the method for training the search space can be called the gradient consistency distillation training method. The training flowchart is as follows: Figure 5 As shown, the training process may include steps S501 to S503:

[0114] S501, sample sub-models from the search space.

[0115] In this step, the search space can be sampled. During the sampling process, three sub-models are sampled. One of the sub-models is the search space itself, called the teacher model. The other two are randomly sampled from the search space and called the student models.

[0116] S502 uses the teacher model to perform knowledge distillation on the student model.

[0117] First, based on the loss between the image processing results of the teacher model and the annotations of the training images, the gradient G of the teacher model is calculated. t-label And save the G of each block. t-label G t-label It can be used to update the gradient of the teacher model. Simultaneously, the gradient G of the student model can be calculated based on the loss between the image processing results of the student model and the annotations of the training images. s-label .

[0118] In this step, the teacher model can be used to perform knowledge distillation on the student model to obtain the gradient of the student model relative to the teacher model, denoted as G. distill For example, when using the teacher model to distill the student model, the gradient G can be calculated based on the loss between the image processing results of the student model and the image processing results of the teacher model. distill Subsequently, by calculating G t-label With G s-label The cosine value of G can be calculated based on formula (8). stu G stu It can be used to update the gradient of the student model. Formula (8) is as follows:

[0119]

[0120] S503 performs gradient updates on the search space.

[0121] In this embodiment, gradient updates of the search space can be achieved by updating the teacher model and the student model using gradients.

[0122] For example, during an update, it can be based on G t-label Gradient updates are performed on the teacher model, and based on G... stu Gradient updates are performed on the student model. The update rule for gradient updates on the student model is shown in formula (8). Formula (8) states: if G... t-label With G s-label The directions are inconsistent, that is, cos(G) t-label G s-label If G < 0, then the student model is updated according to the teacher model. In this case, G stu =G distill Accordingly, if G t-label With G s-label The directions are consistent, that is, cos(G) t-label G s-label If )≥0, then G stu =G s-label+ G distill At this point, it can be based on G s-label+ Gdistill Perform gradient updates on the student model.

[0123] This update method is teacher-centric, which ensures that the gradient updates of the sub-models are in the same direction when updating the gradients of the entire search space, so as to avoid mutual interference between the gradients of the sub-models.

[0124] After completing one round of gradient updates to the search space, the search space obtained from this round of gradient updates can be used as the input for the next round of training. Steps S501 to S503 can then be executed again to perform a new round of gradient updates to the search space. The iteration can stop when the preset number of update rounds has been reached, or when the model convergence condition has been met, and the final target search space can be output.

[0125] Figure 6 This is a schematic diagram illustrating an example of gradient backpropagation in the teacher and student models of this application.

[0126] In the teacher model, Block k-1 and Block k are two adjacent blocks. The solid line between these two adjacent blocks indicates that the feature map obtained by Block k-1 in the teacher model is input into Block k in the teacher model. Similarly, in the student model, Block k-1 and Block k are two adjacent blocks. The solid line between these two adjacent blocks indicates that the feature map obtained by Block k-1 in the student model is input into Block k in the student model.

[0127] exist Figure 6 In the diagram, the dashed line represents the gradient backpropagation process. As an example, suppose Block k in the teacher model is the last block in the teacher model. In this case, Block k in the teacher model can calculate G... t-label The G is passed to Block k-1 in the teacher model. t-label The loss can be calculated based on the image processing result of Block k in the teacher model and the annotation of the training images. Furthermore, Block k-1 in the teacher model can be obtained using G... t-label The gradient G' is obtained by differentiating the weights of Block k-1 in the teacher model. t-label Therefore, when updating the gradient of the teacher model, G can be used. t-label Gradient updates are performed on Block k in the teacher model, and G' is used. t-label Update Block k-1 in the teacher model.

[0128] As an example, suppose Block k in the student model is the last block in the student model. In this case, G can be calculated based on the loss between the image processing result of Block k in the student model and the annotation of the training image. s-label At the same time, it can be determined by G s-label With G t-labe The gradient is determined by whether the cosine value between the given values ​​is less than 0, thus determining the gradient passed to Block k-1 in the student model. Figure 6 As shown, when G s-label With G t-labe When the cosine value between the two values ​​is less than 0, the gradient passed to Block k-1 in the student model is G. distill Otherwise, the gradient passed to Block k-1 in the student model is G. s-label+ G distill Among them, G distill It can be calculated based on the loss between the image processing results of Block k in the student model and the image processing results of Block k in the teacher model.

[0129] For ease of explanation, let's assume the gradient passed to Block k-1 in the student model is G. distill In this case, Block k-1 in the student model can be represented by G. distill The gradient G' is obtained by differentiating the weights of Block k-1 in the student model. distill When updating the gradient of the student model, G can be used. distill Gradient updates are performed on Block k in the student model, and G' is used. distill Update Block k-1 in the student model.

[0130] S303, search for the target model from the target search space.

[0131] In this step, an evolutionary search algorithm can be used to search for the target model from the target search space.

[0132] As an example, a batch of seed networks can first be selected from the target search space. Then, recombination, mutation, evaluation, and selection operations are performed to continuously generate new model architectures. During the selection process, information such as the number of parameters, FLOPs, or hardware latency is introduced to control the lightweight nature of the model. After a certain number of iterations, the final lightweight model, i.e., the target model, is obtained. This target model can be used to perform target task-based image processing on the target image to be processed.

[0133] In evolutionary algorithms, to evaluate the performance of selected seed networks, a pre-trained validation model (precision perceptron) can be used to estimate the precision (e.g., the accuracy of the model output) of each seed network. One possible approach involves training a precision perceptron, which includes: 1) randomly selecting T sub-models from the target search space for validation, obtaining the precision of the T sub-models; 2) combining the model parameters of the T sub-models with their corresponding precisions into a training set; and 3) constructing a perceptron model and training it using this training set to obtain a perceptron that can predict model precision based on the model parameters, i.e., a precision perceptron.

[0134] By employing evolutionary search algorithms to search for target models in the target search space, the workload of model search can be reduced, and the efficiency of searching for target models that meet the deployment conditions can be improved.

[0135] According to the method in this embodiment, a search space with linear complexity is first constructed, which includes operators such as convolution and linear Transformer. Then, a gradient consistency distillation training method is used for model training. This method distills each module in the hybrid convolutional and Transformer structure separately, and addresses the performance degradation caused by mutual interference between different sub-networks. Finally, multiple sub-networks are sampled to train a precision predictor (i.e., a precision perceptron) of the model structure, and an evolutionary search method is used to search for the optimal model under hardware constraints. In this way, a high-precision, lightweight model can be obtained, effectively solving the problem of the difficulty in deploying the searched model on the edge.

[0136] The technical solutions of this application embodiment can produce the following beneficial effects:

[0137] 1) This application proposes a linear complexity search space design method that combines convolution and linear Transformer. By constructing a linear Transformer search space, the final searched model has linear time and space complexity, which can effectively solve the deployment problems of Transformer computation time and storage on mobile devices.

[0138] 2) By proposing a gradient consistency distillation training method, the convolutional part and the Transformer part are distilled in layers. At the same time, gradient consistency is used to effectively solve the interference problem between different sub-models during the search process, thereby improving the accuracy of the searched model.

[0139] 3) The method in this application achieves efficient computation with linear complexity, leverages the global awareness capability of the Transformer, and combines a search method to improve the overall efficiency and accuracy of the model. The model obtained by this method possesses both the lightweight characteristics of being deployable on the edge and the high-precision characteristics of the Transformer.

[0140] 4) During the sampling of seed models, by introducing information such as the number of model parameters, FLOPs, and hardware latency, the lightweight nature of the model can be controlled, and a target model that meets the corresponding deployment conditions can be determined for deployment devices with different hardware resources.

[0141] It is understood that in the embodiments of this application... Figures 1 to 6 The examples provided are merely to facilitate understanding of the embodiments of this application by those skilled in the art, and are not intended to limit the embodiments of this application to the specific scenarios illustrated. Those skilled in the art should refer to these examples. Figures 1 to 6 The examples clearly allow for various equivalent modifications or changes.

[0142] Such modifications or changes also fall within the scope of the embodiments of this application.

[0143] It is also understood that the solutions in the various embodiments of this application can be reasonably combined and used, and

[0144] The explanations or descriptions of the various terms appearing in the embodiments can be referenced or interpreted in each embodiment, and are not limited thereto.

[0145] Based on the foregoing embodiments, this application provides a model search apparatus, which includes...

[0146] Each module can be implemented by a processor in the terminal; of course, it can also be implemented by specific logic circuits. During implementation, the processor can be a central processing unit (CPU), a microprocessor unit (MPU), a digital signal processor (DSP), or a field programmable gate array (FPGA), etc.

[0147] Figure 7 The diagram illustrates the structural composition of a model search apparatus 700 provided in an embodiment of this application. For example... Figure 7 As shown, the device 700 may include:

[0148] Module 701 is used to acquire the search space, which includes the convolution module and the visual transformation module.

[0149] The visual transformation module includes a self-attention submodule for executing a self-attention algorithm, which has a complexity of O(n); and a preprocessing module 702 for using a convolution module to process the labeled training data.

[0150] The training images undergo preprocessing; an image processing module 703 is used to perform target task-based image processing on the preprocessed results using a visual conversion module; a first determination module 704 is used to determine the first gradient of the search space based on the first loss between the image processing results and the annotations of the training images; a gradient update module 705 is used to update the search space based on the first gradient to obtain the target search space; wherein, the target search space is used to search for the target model, and the target model is used to perform target image processing based on the target image to be processed.

[0151] Image processing for the target task.

[0152] In some embodiments, the search space serves as a teacher model, and the apparatus 700 further includes: a second determining module, configured to determine at least one sub-model in the search space as a student model; wherein each student model includes a convolutional sub-module obtained by sampling a convolutional module and a visual transformation sub-module obtained by sampling a visual transformation module; and a distillation module, configured to distill each student model using the teacher model.

[0153] Distillation processing; a third determination module, used to determine the second gradient of each student model based on the results of the distillation processing; and a gradient update module 705, specifically used to update the gradient of the teacher model based on the first gradient.

[0154] The gradient of the corresponding student model is updated based on the second gradient of each student model to obtain the target search space.

[0155] In some embodiments, the distillation module is specifically configured to: for each student model, determine the third gradient of student model 0 based on the second loss between the image processing result of the student model and the image processing result of the teacher model; the third determination module is specifically configured to: for each student model, determine the fourth gradient of the student model based on the third loss between the image processing result of the student model and the annotation of the training image; if the directions of the first gradient and the fourth gradient are consistent, determine the result of adding the third gradient and the fourth gradient as the second gradient of the student model; if the directions of the first gradient and the fourth gradient are inconsistent, determine the third gradient as the second gradient of the student model.

[0156] 5. In some embodiments, the apparatus 700 further includes: a calculation module for each fourth gradient,

[0157] The cosine value between the first gradient and the fourth gradient is calculated; the fourth determination module is used to determine that the directions of the first gradient and the fourth gradient are consistent when the cosine value is greater than or equal to zero; it is also used to determine that the directions of the first gradient and the fourth gradient are inconsistent when the cosine value is less than zero.

[0158] In some embodiments, the second determining module is specifically used for: obtaining a candidate parameter set, the candidate parameter set including model parameters that can be selected by the convolution submodule and model parameters that can be selected by the visual transformation submodule; determining the target model parameters selected by the convolution submodule and the target model parameters selected by the visual transformation submodule from the candidate parameter set; sampling the search space based on the target model parameters selected by the convolution submodule and the target model parameters selected by the visual transformation submodule to obtain the corresponding submodel, and determining the submodel as the student model.

[0159] 5. In some embodiments, the apparatus 700 further includes: a search module for searching for a target model from a target search space based on a search algorithm; wherein the search algorithm includes any one of the following: an evolutionary search algorithm, a random search algorithm, or a reinforcement learning search algorithm.

[0160] In some embodiments, the search algorithm includes an evolutionary search algorithm; the search module is specifically configured to: sample the target search space to obtain at least two seed models, perform a recombination operation and / or a mutation operation on each seed model to obtain at least two offspring models; wherein the recombination operation includes: exchanging at least one pair of model parameters in the model; the mutation operation includes: randomly changing at least one model parameter within a predefined numerical range; using a pre-trained validation model, estimating the accuracy of the image processing results of the at least two offspring models; and determining the target model from the at least two offspring models based on the accuracy of the image processing results of the at least two offspring models.

[0161] In some embodiments, the search module is further configured to: sample the target search space based on model constraints to obtain at least two seed models that satisfy the model constraints; wherein the model constraints include at least one of the following: the number of model parameters is less than or equal to the number of preset parameters; when running the model, the hardware latency is less than or equal to the preset latency; when running the model, the number of floating-point operations per second performed by the hardware is less than or equal to the preset number.

[0162] In some embodiments, the apparatus 700 further includes: a computation module, configured to perform the following computational processing on the preprocessing result of the convolution module using a self-attention submodule: multiplying the first weight matrix with the preprocessing result to obtain a first feature matrix; multiplying the second weight matrix with the preprocessing result to obtain a second feature matrix; sampling the first feature matrix to obtain a third feature matrix; performing a first computational processing on the first feature matrix and the third feature matrix based on a Gaussian kernel function to obtain a first computational result; and performing a second computational processing on the first computational result and the second feature matrix to obtain the output result of the self-attention submodule.

[0163] In some embodiments, the target task includes any one of the following: image classification task, image detection task, and image segmentation task.

[0164] The descriptions of the apparatus embodiments above are similar to those of the method embodiments above, and have similar beneficial effects. In some embodiments, the functions or modules included in the apparatus provided in this application can be used to perform the methods described in the method embodiments above. For technical details not disclosed in the apparatus embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0165] It should be noted that, in the embodiments of this application, if the above-described methods are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, or the parts that contribute to related technologies, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), magnetic disks, or optical disks. Thus, the embodiments of this application are not limited to any specific hardware, software, or firmware, or any combination of hardware, software, and firmware.

[0166] This application embodiment also provides a model search device, including a memory and a processor. The memory stores a computer program that can run on the processor. When the processor executes the program, it implements some or all of the steps in the above method.

[0167] This application also provides a chip, which includes a processor and a communication interface. The processor reads instructions stored in a memory through the communication interface to implement some or all of the steps in the above method. In some embodiments, as an implementation, the chip further includes a memory storing computer programs or instructions. The processor executes the computer programs or instructions stored in the memory. When the computer programs or instructions are executed, the processor performs some or all of the steps in the above method.

[0168] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements some or all of the steps in the above-described method. The computer-readable storage medium may be transient or non-transient.

[0169] This application also provides a computer program including computer-readable code, wherein when the computer-readable code is run in a computer device, a processor in the computer device performs some or all of the steps in the above-described method.

[0170] This application also provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program. When the computer program is read and executed by a computer, it implements some or all of the steps in the above-described method. This computer program product can be implemented specifically through hardware, software, or a combination thereof. In some embodiments, the computer program product is specifically embodied as a computer storage medium; in other embodiments, the computer program product is specifically embodied as a software product, such as a software development kit (SDK), etc.

[0171] It should be noted that the descriptions of the various embodiments above tend to emphasize the differences between them, while their similarities or commonalities can be referred to interchangeably. The descriptions of the above embodiments of the device, storage medium, computer program, and computer program product are similar to the descriptions of the above method embodiments and have similar beneficial effects. For technical details not disclosed in the embodiments of the device, storage medium, computer program, and computer program product of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0172] It should be noted that, Figure 8 This is a schematic diagram of a hardware entity of the model search device in an embodiment of this application, such as... Figure 8 As shown, the hardware entity of the model search device 800 includes: a processor 801, a communication interface 802, and a memory 803, wherein:

[0173] The processor 801 typically controls the overall operation of the model search device 800.

[0174] The communication interface 802 enables the model search device 800 to communicate with other terminals or servers via a network.

[0175] The memory 803 is configured to store instructions and applications executable by the processor 801, and can also cache data to be processed or already processed by the various modules in the processor 801 and the speech model search 800 (e.g., image data, audio data, voice communication data, and video communication data). It can be implemented using flash memory or random access memory (RAM). Data transfer between the processor 801, the communication interface 802, and the memory 803 can be performed via bus 804.

[0176] The model search device 800 can implement the corresponding processes in the various methods of the embodiments of this application, which will not be described in detail here for the sake of brevity.

[0177] It should be understood that the phrase "one embodiment" or "an embodiment" throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of this application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. It should be understood that in the various embodiments of this application, the sequence numbers of the above steps / processes do not imply a sequential order of execution; the execution order of each step / process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. The sequence numbers of the above embodiments of this application are merely descriptive and do not represent the superiority or inferiority of the embodiments.

[0178] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0179] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.

[0180] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units. They may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0181] In addition, each functional unit in the various embodiments of this application can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0182] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as mobile storage devices, read-only memory (ROM), magnetic disks, or optical disks.

[0183] Alternatively, if the integrated units described above are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence or the part that contributes to related technologies, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, magnetic disks, or optical disks.

[0184] The above description is merely an embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A method for model search, characterized in that, The method includes: A search space is obtained, comprising a convolutional module and a visual transformation module; wherein the visual transformation module includes a self-attention submodule for executing a self-attention algorithm, the self-attention algorithm having a complexity of O(n). O ( n ); The convolution module is used to preprocess the labeled training images; The visual conversion module is used to perform target task-based image processing on the preprocessed results; The first gradient of the search space is determined based on the first loss between the result of the image processing and the annotation of the training image; The search space serves as a teacher model, and the method further includes: At least one sub-model in the search space is identified as the student model; For each student model, a third gradient of the student model is determined based on a second loss between the image processing results of the student model and the image processing results of the teacher model. For each student model, a fourth gradient of the student model is determined based on a third loss between the image processing result of the student model and the annotation of the training image; if the directions of the first gradient and the fourth gradient are consistent, the result of adding the third gradient and the fourth gradient is determined as the second gradient of the student model; if the directions of the first gradient and the fourth gradient are inconsistent, the third gradient is determined as the second gradient of the student model. The teacher model is updated based on the first gradient, and the corresponding student model is updated based on the second gradient of each student model to obtain the target search space; wherein, the target search space is used to search for the target model, and the target model is used to perform image processing based on the target task on the target image to be processed.

2. The method according to claim 1, characterized in that, Each student model includes a convolutional submodule obtained by sampling the convolutional module, and a visual transformation submodule obtained by sampling the visual transformation module.

3. The method according to claim 1 or 2, characterized in that, The method further includes: For each of the fourth gradients, determine the cosine value between the first gradient and the fourth gradient; If the cosine value is greater than or equal to zero, the first gradient is determined to be in the same direction as the fourth gradient. If the cosine value is less than zero, it is determined that the directions of the first gradient and the fourth gradient are inconsistent.

4. The method according to claim 1 or 2, characterized in that, The step of identifying at least one sub-model in the search space as a student model includes: Obtain a candidate parameter set, which includes model parameters that can be selected by the convolution submodule and model parameters that can be selected by the visual transformation submodule; From the candidate parameter set, determine the target model parameters selected by the convolution submodule and the target model parameters selected by the visual transformation submodule; Based on the target model parameters selected by the convolution submodule and the target model parameters selected by the visual transformation submodule, the search space is sampled to obtain the corresponding submodel, and the submodel is determined as the student model.

5. The method according to claim 1 or 2, characterized in that, The method further includes: The target model is obtained from the target search space based on the search algorithm; wherein the search algorithm includes any one of the following: evolutionary search algorithm, random search algorithm, and reinforcement learning search algorithm.

6. The method according to claim 5, characterized in that, The search algorithm includes an evolutionary search algorithm; the step of searching for the target model from the target search space based on the search algorithm includes: The target search space is sampled to obtain at least two seed models. Recombination and / or mutation operations are performed on each seed model to obtain at least two offspring models. The recombination operation includes: exchanging at least one pair of model parameters in the model; the mutation operation includes: randomly changing at least one model parameter within a predefined numerical range. Using a pre-trained validation model, the accuracy of the image processing results of the at least two child models is estimated; The target model is determined from the at least two child models based on the accuracy of the image processing results of the at least two child models.

7. The method according to claim 6, characterized in that, The sampling of the target search space yields at least two seed models, including: The target search space is sampled based on model constraints to obtain at least two seed models that satisfy the model constraints; wherein the model constraints include at least one of the following: The number of model parameters is less than or equal to the preset number of parameters; when running the model, the hardware latency is less than or equal to the preset latency; when running the model, the number of floating-point operations performed by the hardware per second is less than or equal to the preset number.

8. The method according to claim 1 or 2, further comprising: The preprocessing results of the convolution module are processed using the self-attention submodule as follows: Multiply the first weight matrix by the preprocessing result to obtain the first feature matrix; Multiply the second weight matrix by the preprocessing result to obtain the second feature matrix; The first feature matrix is ​​sampled to obtain the third feature matrix; A first operation is performed on the first feature matrix and the third feature matrix based on the Gaussian kernel function to obtain a first operation result; The first operation result and the second feature matrix are subjected to a second operation to obtain the output result of the self-attention submodule.

9. The method according to claim 5, characterized in that, The target task includes any one of the following: Image classification tasks, image detection tasks, and image segmentation tasks.

10. A model search apparatus, characterized in that, The device includes: An acquisition module is used to acquire the search space, which includes a convolution module and a visual transformation module; wherein, the visual transformation module includes a self-attention submodule for executing a self-attention algorithm, the self-attention algorithm having a complexity of O(n log n). O ( n ); A preprocessing module is used to preprocess the labeled training images using the convolution module; The image processing module is used to perform target task-based image processing on the preprocessed result using the visual conversion module; The first determining module is used to determine the first gradient of the search space based on the first loss between the result of the image processing and the annotation of the training image; The search space serves as a teacher model, and the device further includes: The second determining module is used to determine at least one sub-model in the search space as a student model; The distillation module is used to: for each student model, determine the third gradient of the student model based on a second loss between the image processing results of the student model and the image processing results of the teacher model; The third determining module is configured to: for each student model, determine the fourth gradient of the student model based on the third loss between the image processing result of the student model and the annotation of the training image; when the directions of the first gradient and the fourth gradient are consistent, determine the result of adding the third gradient and the fourth gradient as the second gradient of the student model; when the directions of the first gradient and the fourth gradient are inconsistent, determine the third gradient as the second gradient of the student model. The gradient update module is used to: update the gradient of the teacher model based on the first gradient, and update the gradient of the corresponding student model based on the second gradient of each student model to obtain the target search space; wherein, the target search space is used to search for the target model, and the target model is used to perform image processing based on the target task on the target image to be processed.

11. A model search device, characterized in that, The model search device includes: Memory is used to store executable instructions for a computer; A processor, connected to the memory, is configured to implement the method of any one of claims 1 to 9 by executing the computer-executable instructions.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by at least one processor, implements the method as described in any one of claims 1 to 9.

Images