Visual model generation method and system, electronic equipment and storage medium

Abstract

The invention provides a visual model generation method and system, electronic equipment and a storage medium, and the method comprises the steps: receiving a generation instruction of a target visual task model, the generation instruction comprising a target task demand of the target visual task model and a target data set feature of the target visual task model; determining candidate visual task teacher models from a plurality of preset visual task teacher models according to the target task requirements and the target data set features; determining a target attention weight of the candidate visual task teacher model based on an attention mechanism; and generating a target visual task model according to the target attention weight and the candidate visual task teacher model. Through the embodiment of the invention, the problem that teacher model selection is relatively fixed in the existing method can be solved. And meanwhile, the advantages of different teacher models are fully utilized, and the model performance and the adaptability to different tasks are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of visual models, and in particular to a method for generating a visual model, a system for generating a visual model, an electronic device and a computer-readable storage medium. Background Art

[0002] In the field of computer vision, visual models play a key role in many downstream tasks. With the development of technology, the performance and efficiency requirements of visual models are getting higher and higher. For example, in intelligent security monitoring, it is necessary to accurately identify the target, behavior, and scene information in the image to ensure public safety; in the field of medical image analysis, medical images need to be accurately semantically segmented and classified to assist doctors in disease diagnosis; in autonomous driving technology, vehicles need to perceive and understand the surrounding environment in real time to ensure driving safety. These application scenarios all rely on high-performance visual models to provide accurate visual information processing capabilities.

[0003] Existing methods are often limited to a few specific popular models when selecting teacher models. This fixed selection method limits the improvement of model performance. Summary of the invention

[0004] In view of the above problems, a method for generating a visual model, a system for generating a visual model, an electronic device and a computer-readable storage medium are proposed to overcome the above problems or at least partially solve the above problems, including:

[0005] A method for generating a visual model, the method comprising:

[0006] Receiving a generation instruction of a target visual task model, the generation instruction including a target task requirement of the target visual task model and a target data set feature of the target visual task model;

[0007] Determining a candidate visual task teacher model from a plurality of preset visual task teacher models according to the target task requirements and the target data set characteristics;

[0008] Determining a target attention weight of the candidate visual task teacher model based on an attention mechanism;

[0009] A target visual task model is generated according to the target attention weight and the candidate visual task teacher model.

[0010] Optionally, the method further comprises:

[0011] Collect multiple visual task teacher models, and evaluate the multiple visual task teacher models on multiple standard data sets to obtain performance indicators of each visual task teacher model corresponding to different task requirements;

[0012] and, determine the structural characteristics of the teacher models for each visual task;

[0013] The step of determining a candidate visual task teacher model from a plurality of preset visual task teacher models according to the target task requirements and the target data set characteristics includes:

[0014] Determine the target performance indicators corresponding to each visual task teacher model under the target task requirements; and determine the target structural features that meet the characteristics of the target data set;

[0015] The candidate visual task teacher model is determined from the multiple visual task teacher models according to the target performance indicator and the target structural feature.

[0016] Optionally, the target loss function of the target visual task model includes a basic loss function and a task-specific adjustment term;

[0017] The task-specific adjustment term in the target loss function is determined based on the target task requirements.

[0018] Optionally, the method further comprises:

[0019] When training the target visual task model, the target attention weight is adjusted based on the performance indicator corresponding to the current target attention weight.

[0020] Optionally, the method further comprises:

[0021] When training the target visual task model, the model parameters of the target visual task model are adjusted based on performance indicators corresponding to current model parameters of the target visual task model.

[0022] Optionally, the method further comprises:

[0023] When training the target visual task model, the model architecture of the target visual task model is adjusted based on the performance indicator corresponding to the current model architecture of the target visual task model.

[0024] The embodiment of the present invention further provides a system for generating a visual model, characterized in that the system comprises:

[0025] The teacher model selection and fusion module is used to receive a generation instruction of a target visual task model, wherein the generation instruction includes a target task requirement of the target visual task model and a target data set feature of the target visual task model; determine a candidate visual task teacher model from a plurality of preset visual task teacher models according to the target task requirement and the target data set feature; determine a target attention weight of the candidate visual task teacher model based on an attention mechanism; and generate a target visual task model according to the target attention weight and the candidate visual task teacher model.

[0026] Optionally, the system further comprises:

[0027] An adaptive loss function module, used to determine a target loss function of the target visual task model, wherein the target loss function includes a basic loss function and a task-specific adjustment item;

[0028] The hybrid architecture dynamic optimization module is used to generate a model architecture of the target visual task model.

[0029] An embodiment of the present invention further provides an electronic device, comprising a processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein the computer program implements the above-mentioned method for generating a visual model when executed by the processor.

[0030] An embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the method for generating a visual model as described above is implemented.

[0031] The embodiments of the present invention have the following advantages:

[0032] In an embodiment of the present invention, a generation instruction of a target visual task model is received, and the generation instruction includes the target task requirements of the target visual task model and the target data set characteristics of the target visual task model; according to the target task requirements and the target data set characteristics, a candidate visual task teacher model is determined from a plurality of preset visual task teacher models; based on the attention mechanism, the target attention weight of the candidate visual task teacher model is determined; according to the target attention weight and the candidate visual task teacher model, the target visual task model is generated. Through the embodiment of the present invention, when facing a specific visual task, the most suitable teacher model can be dynamically screened out according to the task requirements and the data set characteristics for combination, thereby breaking through the problem of relatively fixed selection of teacher models in the existing methods. At the same time, through the fusion method based on the attention mechanism, the influence weight of the teacher model on the student model can be dynamically adjusted according to the relationship between the teacher model and the student model, making full use of the advantages of different teacher models, and improving the model performance and adaptability to different tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the technical solution of the present invention, the accompanying drawings required for use in the description of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0034] Figure 1 is a flowchart of a method for generating a visual model according to an embodiment of the present invention;

[0035] Figure 2 is a flowchart of another method for generating a visual model according to an embodiment of the present invention;

[0036] Figure 3 is a schematic diagram of an overall framework of an embodiment of the present invention;

[0037] Figure 4 It is a structural schematic diagram of a visual model generation system according to an embodiment of the present invention. DETAILED DESCRIPTION

[0038] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0039] Existing methods are often limited to a few specific popular models when selecting teacher models, lacking the flexibility to dynamically select according to specific task requirements and dataset characteristics. This fixed selection method may result in the inability to fully utilize the advantages of the most suitable teacher model on certain specific tasks or datasets, limiting the improvement of model performance.

[0040] In order to break through the limitation of relatively fixed selection of teacher models in existing methods and improve model performance, the embodiment of the present invention proposes a method for generating a visual model by dynamically selecting a teacher model based on task requirements and data set characteristics. Figure 1 , shows a flowchart of a method for generating a visual model according to an embodiment of the present invention, which may include the following steps:

[0041] Step 101: Receive a generation instruction of a target visual task model, where the generation instruction includes a target task requirement of the target visual task model and a target data set feature of the target visual task model.

[0042] Among them, the target visual task model may refer to a model used to complete a visual task, and the visual task may include image classification, semantic segmentation, object detection, etc.

[0043] In an embodiment of the present invention, when a user needs to complete a target visual task model for a specific task, he or she may first input a generation instruction for the target visual task model into the visual model generation system; the generation instruction may include the target task requirements of the target visual task model and the target data set features of the target visual task model.

[0044] The target task requirement may refer to the requirement of a specific task corresponding to the target visual task model. For example, the target task requirement may be "the application scenario is the semantic segmentation task of medical images". The target task requirement may also include specific requirements of the task, such as requirements for accuracy, speed, etc.

[0045] The target dataset features of the target visual task model may refer to the features of the dataset it processes, for example, medical image features have microstructures.

[0046] In some feasible embodiments, after receiving a generation instruction for a target visual task model, the visual model generation system may first extract the target task requirements of the target visual task model and the target data set features of the target visual task model from the generation instruction.

[0047] Step 102: Determine a candidate visual task teacher model from a plurality of preset visual task teacher models according to the target task requirements and the target data set characteristics.

[0048] After obtaining the target task requirements of the target visual task model and the target data set characteristics of the target visual task model, the visual task model generation system can first determine one or more candidate visual task teacher models that can meet the target task requirements and the target data set characteristics from multiple preset visual task teacher models based on the target task requirements and the target data set characteristics.

[0049] Among them, the visual task teacher model is a model used to guide and assist visual task learning. It aims to help the student model improve its performance by imitating the behavior of the teacher model.

[0050] Step 103: Based on the attention mechanism, determine the target attention weight of the candidate visual task teacher model.

[0051] After determining one or more candidate visual task teacher models, the visual task generation system can determine the target attention weights of each candidate visual task teacher model based on the attention mechanism, so as to fully utilize the advantages of different teacher models and improve the model performance and adaptability to different tasks.

[0052] Step 104: Generate a target visual task model based on the target attention weight and the candidate visual task teacher model.

[0053] After determining the target attention weight, the visual task generation system can generate a target visual task model corresponding to the generation instruction based on the target attention weight and the candidate visual task teacher model.

[0054] In practical applications, a training data set and a verification data set corresponding to the target visual task model can be prepared to ensure the quality and representativeness of the data set; among them, for different visual tasks, the visual model generation system can select different data sets to train the target visual task model, and the embodiments of the present invention are not limited to this.

[0055] In an embodiment of the present invention, a generation instruction of a target visual task model is received, and the generation instruction includes the target task requirements of the target visual task model and the target data set characteristics of the target visual task model; according to the target task requirements and the target data set characteristics, a candidate visual task teacher model is determined from a plurality of preset visual task teacher models; based on the attention mechanism, the target attention weight of the candidate visual task teacher model is determined; according to the target attention weight and the candidate visual task teacher model, the target visual task model is generated. Through the embodiment of the present invention, when facing a specific visual task, the most suitable teacher model can be dynamically screened out according to the task requirements and the data set characteristics for combination, thereby breaking through the problem of relatively fixed selection of teacher models in the existing methods. At the same time, through the fusion method based on the attention mechanism, the influence weight of the teacher model on the student model can be dynamically adjusted according to the relationship between the teacher model and the student model, making full use of the advantages of different teacher models, and improving the model performance and adaptability to different tasks.

[0056] Reference Figure 2 , shows a flowchart of another method for generating a visual model according to an embodiment of the present invention, which may include the following steps:

[0057] Step 201: collect multiple visual task teacher models, and evaluate the multiple visual task teacher models on multiple standard data sets to obtain performance indicators of each visual task teacher model corresponding to different task requirements; and determine the structural characteristics of each visual task teacher model.

[0058] In some feasible embodiments, a model performance and structural feature database may be established in advance.

[0059] Specifically, multiple visual task teacher models can be collected first, including but not limited to traditional convolutional neural network (CNN) models, and Transformer-based models that have been popular in recent years, such as CLIP (Contrastive Language-Image Pretraining), DINOv2 (Deep Image Network Optimization v2), Llama3.2-11B (Large Language Model Architecture 3.2-11 Billion parameters), Llama3.2-90B (Large Language Model Architecture 3.2-90 Billion parameters), etc., and the embodiments of the present invention do not limit this.

[0060] After collecting multiple visual task teacher models, these visual task teacher models can be evaluated on multiple standard datasets.

[0061] Among them, standard datasets can cover the ImageNet dataset for image classification, the ADE20k dataset for semantic segmentation, and the COCO (Common Objects in Context) dataset for target detection.

[0062] In the process of evaluating multiple visual task teacher models using standard datasets, the performance indicators of each model on different tasks can be recorded in detail, such as accuracy, recall, mIOU (Mean Intersection over Union), etc.

[0063] In addition, the visual model generation system can also analyze each visual task teacher model to determine the structural characteristics of each visual task teacher model.

[0064] Exemplarily, the structural features of the visual task teacher model may include information such as: the number of layers of the model, the number of neurons in each layer, the size of the convolution kernel, the number of attention heads of the Transformer layer, etc., which is not limited in this embodiment of the present invention.

[0065] After obtaining the performance indicators of each visual task teacher model corresponding to different task requirements, as well as the structural features of each visual task teacher model, these performance indicators and structural features can be stored in a specially established model performance and structural feature database for subsequent query and use.

[0066] Step 202: Receive a generation instruction for a target visual task model.

[0067] When a user needs to complete a target visual task model for a specific task, he or she may first input generation instructions for the target visual task model into the visual model generation system.

[0068] After receiving a generation instruction for a target visual task model, the visual model generation system may first extract target task requirements of the target visual task model and target data set features of the target visual task model from the generation instruction.

[0069] Step 203: determine the target performance indicators corresponding to each visual task teacher model under the target task requirements; and determine the target structural features that meet the characteristics of the target data set.

[0070] After determining the target task requirements of the target visual task model and the target data set characteristics of the target visual task model, the visual model generation system can screen the visual task teacher model based on this information.

[0071] Specifically, the visual model generation system can first determine the target performance indicators corresponding to each visual task teacher model in the database under the target task requirements.

[0072] In addition, the visual model generation system can also determine the target structural features that meet the characteristics of the target data set from the structural features stored in the database.

[0073] Step 204: Determine a candidate visual task teacher model from a plurality of visual task teacher models according to the target performance indicator and the target structural feature.

[0074] Based on the determined target performance indicators and target structural features, one or more corresponding visual task teacher models can be indexed from multiple visual task teacher models in the database as candidate visual task teacher models.

[0075] Step 205: Based on the attention mechanism, determine the target attention weight of the candidate visual task teacher model.

[0076] After determining one or more candidate visual task teacher models, the visual task generation system can determine the target attention weights of each candidate visual task teacher model based on the attention mechanism, so as to fully utilize the advantages of different teacher models and improve the model performance and adaptability to different tasks.

[0077] Step 206: Generate a target visual task model based on the target attention weight and the candidate visual task teacher model.

[0078] After determining the target attention weight, the visual task generation system can generate a target visual task model corresponding to the generation instruction based on the target attention weight and the candidate visual task teacher model.

[0079] In some feasible embodiments, the corresponding layer outputs of the student model (i.e., the target visual task model) and the visual task teacher model can be input into a small neural network to calculate the target attention weight.

[0080] Then, the outputs of the visual task teacher models can be weighted summed according to the target attention weights to obtain the fused output. This fusion method can better utilize the advantages of different visual task teacher models and improve the performance of the student model.

[0081] In an embodiment of the present invention, the above method may further include the following steps:

[0082] When training the target visual task model, the target attention weight is adjusted based on the performance indicator corresponding to the current target attention weight.

[0083] In some feasible embodiments, during the process of fusing candidate visual task teacher models based on the attention mechanism, the visual model generation system can detect the effect of the fusion and evaluate the quality of the fusion by detecting the performance indicators of the student model on the validation set.

[0084] If the fusion effect is not good, the visual model generation system can adjust the parameters of the attention mechanism, or reselect the candidate visual task teacher model until the preset fusion effect is achieved. The preset fusion effect can be set according to actual needs, and the embodiment of the present invention does not limit this.

[0085] In one embodiment of the present invention, the target loss function of the target visual task model includes a basic loss function and a task-specific adjustment term;

[0086] The task-specific adjustments in the objective loss function are determined based on the target task requirements.

[0087] In practical applications, traditional loss functions usually use a fixed weight distribution method when dealing with the fusion of multiple visual task teacher models, and cannot dynamically adjust the weights according to the performance of the visual task teacher model at different training stages and on different tasks. At the same time, for different visual tasks (such as image classification, semantic segmentation, object detection, etc.), the existing loss functions lack targeted design and cannot adapt well to the characteristics of various tasks, thus affecting the performance of the model on different tasks.

[0088] In view of this, in an embodiment of the present invention, the target loss function for the target visual task model may include a basic loss function and task-specific adjustment items; wherein the task-specific adjustment items in the target loss function are determined based on the target task requirements.

[0089] Specifically, the objective loss function consists of a base loss function and a task-specific adjustment term.

[0090] Among them, the basic loss function can adopt a combination of improved cosine distance loss and Smooth-L1 loss, which can better measure the degree of feature matching between the teacher model and the student model of the visual task. Task-specific adjustments are designed for different visual tasks, such as considering the differences in target boundaries and regions in semantic segmentation tasks, and considering category confusion in image classification tasks.

[0091] In addition, through the dynamic weight adjustment mechanism, the weight of the visual task teacher model in the loss function is updated in real time according to the performance of the visual task teacher model in different training stages and tasks, so that the loss function can better adapt to the characteristics of different tasks and visual task teacher models, thereby improving model performance.

[0092] For feature matching of the visual task teacher model, the cosine distance loss between feature vectors is first calculated to measure the similarity of features. At the same time, the Smooth-L1 loss is introduced to deal with the numerical differences between feature vectors, making the loss function more robust when dealing with features of different scales.

[0093] For task-specific adjustments, in semantic segmentation tasks, loss adjustments based on object boundaries and regions are designed. Object boundaries refer to the dividing lines or contours between different semantic categories in an image. It represents the clear boundaries between different objects or regions.

[0094] Object boundaries are very important in semantic segmentation because they help the model accurately identify and distinguish different objects or regions. Good object boundaries can improve the accuracy and visual quality of segmentation results.

[0095] For example, in an image containing a person and a background, the target boundary is the dividing line between the person and the background. The model needs to accurately identify this boundary to ensure that the person is correctly segmented and the background is excluded.

[0096] A region is a collection of continuous pixels in an image that have the same semantic category. All pixels in the region are assigned the same category label.

[0097] Region is the basic unit of semantic segmentation, which represents the distribution of different semantic categories in the image. The model needs to segment the image into multiple regions, each of which corresponds to a specific semantic category.

[0098] For example, in an image containing a person and a background, the region can be divided into two parts: one is the region of the person, and the other is the region of the background. The model needs to correctly assign all pixels in the image to these two regions.

[0099] The loss function is adjusted by calculating the difference between the predicted segmentation result and the true segmentation result on the target boundary, as well as the error in different areas (such as inside the target, background, etc.). Suppose the predicted segmentation result is The true segmentation result is Y, then the loss adjustment term L based on the target boundary and area seg It can be expressed as:

[0100]

[0101] Where M is the number of samples, α and β are balance parameters, and l boundary and l region They are functions for calculating target boundary and area losses respectively.

[0102] In the visual task of image classification, a loss adjustment item based on category confusion is added. According to the confusion between different categories in the classification results, the penalty intensity of the loss function for misclassification of different categories is dynamically adjusted. Suppose the total number of categories is C, and the true label of the jth category on the i-th sample is y ij , the predicted label is The loss adjustment term based on category confusion can be expressed as:

[0103]

[0104] Where N is the total number of samples, γ ij is the weight dynamically adjusted according to the category confusion, l confusion A function for calculating category confusion loss.

[0105] In the visual task of object detection, semantic segmentation loss can be introduced to improve the model's ability to recognize the boundaries of target objects. Common semantic segmentation loss functions include cross entropy loss and Dice loss. Instance segmentation loss can also be introduced to improve the model's ability to recognize the internal area of ​​the target object. Common instance segmentation loss functions include cross entropy loss and Dice loss. Bounding box regression loss can also be introduced to improve the model's ability to accurately locate the bounding box of the target object. Common bounding box regression loss functions include SmoothL1 loss.

[0106] In some feasible embodiments, a weight adjustment model may be established that can dynamically adjust the weights of various visual task teacher models in the loss function according to the performance of the visual task teacher models at different training stages and on different tasks.

[0107] During the training process, the performance of the visual task teacher model is regularly evaluated, and the accuracy, recall and other indicators are calculated on the validation set. Let the performance indicator of the teacher model k at the current stage be P k , the performance index of the previous stage is P k-1 , the weight of the previous stage is w k-1 , the weight adjustment model uses a gradient descent-based algorithm to update the weights w of the candidate visual task teacher model k , the update formula is as follows:

[0108]

[0109] Among them, η is the learning rate, λ is the performance impact factor, and L is the total loss function. If a candidate visual task teacher model contributes more to the improvement of task performance at a certain stage, its weight will increase accordingly; conversely, if its contribution is small, its weight will decrease.

[0110] In an embodiment of the present invention, the above method may further include the following steps:

[0111] When training the target visual task model, the model parameters of the target visual task model are adjusted based on the performance indicators corresponding to the current model parameters of the target visual task model.

[0112] In an embodiment of the present invention, the above method may further include the following steps:

[0113] When training a target visual task model, the model architecture of the target visual task model is adjusted based on performance indicators corresponding to the current model architecture of the target visual task model.

[0114] In practical applications, the current high-performance architecture is generally a hybrid architecture based on CNN-Transformer. However, the current hybrid architecture (such as the CNN-Transformer-based architecture) is relatively fixed in design. Once the architecture parameters are determined, it is difficult to flexibly adjust according to the characteristics of the input data (such as image resolution, target size and distribution, etc.) and task requirements (such as speed and accuracy requirements) during training and application. This makes it impossible for the model to achieve the optimal performance state when facing different tasks and data sets, limiting the versatility and adaptability of the model.

[0115] In view of this, in the embodiment of the present invention, in the design of the hybrid architecture of the student model (i.e., the target visual task model), the initial structure and parameters of the CNN layer and the Transformer layer of the target visual task model are first determined according to the input data characteristics and task requirements. Then, during the training process, the parameters of the CNN layer and the Transformer layer are dynamically adjusted according to the performance of the target visual task model on the validation set and its adaptability to different tasks. A reinforcement learning algorithm is used to learn the best architecture adjustment strategy, so that the target visual task model can automatically adjust to the optimal architecture state according to the characteristics of different tasks and data sets, thereby improving the performance of the target visual task model on different tasks and data sets, and enhancing the versatility and adaptability of the model.

[0116] Specifically, the initial architecture can be determined first; first, the characteristics of the input data and the task requirements can be analyzed to determine the initial structure and parameters of the CNN layer and Transformer layer of the target visual task model. If the input data is a high-resolution image and the task is semantic segmentation, the initial CNN layer uses a larger convolution kernel and more convolution layers to better extract local features; the Transformer layer sets more attention heads and appropriate window sizes to adapt to targets of different scales.

[0117] For low-resolution image classification tasks, the CNN layer uses smaller convolution kernels and shallower layers to reduce the amount of computation; the Transformer layer optimizes the global attention mechanism to improve classification accuracy.

[0118] During the training process, the convolution kernel size, step size, padding and other parameters of the CNN layer, as well as the number of attention heads, window size, number of layers and other parameters of the Transformer layer are dynamically adjusted according to the performance of the target visual task model on the validation set and its adaptability to different tasks.

[0119] If the target visual task model does not perform well in segmenting small objects in the semantic segmentation task, the number of convolution kernels in the CNN layer can be increased and the step size can be reduced to enhance the feature extraction capability of small objects. At the same time, the attention of the Transformer layer can be adjusted to improve the performance of the model on different tasks and datasets.

[0120] A policy optimization algorithm based on reinforcement learning is used to learn the best architecture adjustment strategy. The state space is defined as the performance indicators and architecture parameters of the target visual task model at different training stages, and the action space is the possible architecture adjustment operations (such as increasing or decreasing the number of convolution kernels, changing the number of attention heads, etc.).

[0121] According to the performance change of the target visual task model on the validation set, corresponding rewards or penalties are given. If the model performance is improved, a positive reward r is given. + ; If performance degrades, give a negative reward r - . Then the goal of reinforcement learning is to maximize the cumulative reward R, and its expectation can be expressed as:

[0122]

[0123] Among them, γ is the discount factor, r t is the reward obtained at time step t. Through continuous experimentation and adjustment, the algorithm eventually learns the best architecture adjustment strategy, so that the target visual task model can automatically adjust to the best architecture state according to different tasks and data set characteristics.

[0124] For example, Figure 3 As shown, the generation of the visual model includes the data preparation stage, the teacher model selection and fusion stage, and the model training distillation stage.

[0125] The first is the data preparation stage:

[0126] Comprehensive collection of visual task teacher models, including classic CNN models and emerging Transformer-based models (Llama3.2-11B, Llama3.2-90B, CLIP, etc.). These models are evaluated on standard datasets, and detailed performance indicators and structural feature information are recorded and stored in the database.

[0127] Determine the task requirements based on the specific application scenario, such as image classification, semantic segmentation or target detection tasks, as well as the specific requirements of the task, such as requirements for accuracy and speed.

[0128] Prepare corresponding training and validation datasets to ensure the quality and representativeness of the datasets. Choose different datasets for different tasks.

[0129] Teacher model selection and fusion stage:

[0130] First, candidate visual task teacher models that meet the conditions (target task requirements and target dataset characteristics) are screened out from the database, and then the candidate visual task teacher models are fused using the attention mechanism.

[0131] In the process of fusion of candidate visual task teacher models based on the attention mechanism, the performance indicators of the student model on the validation set are tested to evaluate the quality of fusion. If the fusion effect is not good, the parameters of the attention mechanism can be adjusted, or the candidate visual task teacher model can be reselected until a satisfactory fusion effect is achieved.

[0132] Model training distillation stage:

[0133] Initialize training and train the fused model. First, set the basic parameters of training, including the number of training rounds, batch size, learning rate, etc. Initialize the parameters of the fused model and randomly assign values ​​within a reasonable range. At the same time, initialize the relevant parameters in the adaptive loss function, including the weight of the basic loss function, the parameters of the task-specific adjustment items, and the relevant parameters of the dynamic weight adjustment mechanism.

[0134] During the training process, in each round of training, sample data is extracted from the training data set according to the batch size B, including input data X and corresponding label Y. Input data X is input into the fused model, and the output of the model is obtained after the forward propagation calculation of the model.

[0135] The adaptive loss function module designs a loss function framework based on multi-task learning, which consists of a basic loss function and multiple task-specific adjustment items. The loss value is calculated according to the set loss function framework. First, the value of the basic loss function is calculated. Suppose the set of candidate visual task teacher models is T = {T1, T2, ..., T n}, for each candidate visual task teacher model T i , calculate the feature matching loss L between it and the student model on the current sample i，base , for example, for the basic loss function using the improved cosine distance loss and Smooth-L1 loss combination, L i，base The calculation can be expressed as:

[0136]

[0137] Among them, α0 and β0 are the weights of the two losses in the basic loss function. is the output of the student model, T i For the candidate visual task teacher model T i The corresponding output of .

[0138] Then calculate the loss value of the task-specific adjustment term, such as in the semantic segmentation task, calculate the loss adjustment term L based on the object boundary and area seg (as defined above), in the image classification task, the loss adjustment term L based on the class confusion is calculated cls (As defined above). Finally, the base loss function loss and the task-specific adjustment loss are added together to get the total loss L:

[0139]

[0140] According to the total loss value L, the gradient of the model parameters is calculated using the back propagation algorithm. During the back propagation process, the adaptive weight adjustment mechanism dynamically adjusts the weights of each candidate visual task teacher model in the loss function according to the performance of the candidate visual task teacher model in the current round and the current task. Suppose the performance index of the candidate visual task teacher model k in the current round is P k (obtained by calculating the accuracy on the validation set), the weight of the previous round is w k-1 , then the weight adjustment model uses a gradient descent-based algorithm to update the weights w of the candidate visual task teacher model k , the update formula is as follows:

[0141]

[0142] Where η is the learning rate and λ is the performance influencing factor. If a candidate visual task teacher model contributes more to the improvement of task performance in the current round, its weight will increase accordingly; conversely, if its contribution is small, its weight will decrease.

[0143] According to the calculated gradient, the optimizer is used to update the model parameters. When using the Adam optimizer, the formula for updating the model parameters θ according to the current gradient and learning rate is:

[0144] m t =β1·m t-1 +(1-β1)·g t

[0145]

[0146] Among them, m t and v t are the first-order moment estimate and the second-order moment estimate of the tth step, respectively, g t is the current gradient, β1 and β2 are the hyperparameters of the optimizer, and ε is a small positive number used to prevent the denominator from being zero.

[0147] Dynamic adjustment of hybrid architecture:

[0148] During the training process, after the hybrid architecture dynamic optimization module determines the initial architecture, it dynamically adjusts the convolution kernel size, step size, padding and other parameters of the CNN layer, as well as the number of attention heads, window size, number of layers and other parameters of the Transformer layer based on the performance of the student model on the validation set and its adaptability to different tasks.

[0149] After each K rounds of training, the performance of the model on the validation set is evaluated and relevant performance indicators (such as accuracy, mIOU, etc.) are calculated. Let the performance indicator on the current validation set be P val , the performance index of the last evaluation was if (δ is the set performance improvement threshold), then the adjustment amount of the CNN layer and Transformer layer parameters is calculated according to the current performance and task requirements.

[0150] For the convolution kernel size k of the CNN layer, the calculation of the adjustment amount Δk can be expressed as:

[0151]

[0152] Among them, μ is an adjustment coefficient, which is related to the task and data characteristics.

[0153] For the number of attention heads h in the Transformer layer, the calculation of the adjustment Δh can be expressed as:

[0154]

[0155] Among them, v is an adjustment coefficient, which is related to the task and data characteristics.

[0156] According to the calculated adjustment amount, update the parameters of the CNN layer and Transformer layer:

[0157] k new =k+Δk

[0158] h new =h+Δh

[0159] And so on, update other parameters. Through this dynamic adjustment method, the model can continuously optimize its own architecture according to the characteristics of different tasks and datasets, and improve the performance of the model on different tasks and datasets.

[0160] In the embodiment of the present invention, a model database is established to record the performance and structural characteristics of multiple large visual models, the teacher model is dynamically selected according to the task requirements and data set characteristics, and the influence of the teacher model on the student model is integrated by the attention mechanism. The adaptability of the model to different visual tasks is improved, and the advantages of different teacher models can be fully utilized to enhance the performance of the student model.

[0161] Construct a loss function framework that includes a basic loss function and task-specific adjustment items, design specific loss adjustment items for different tasks, and establish a dynamic weight adjustment mechanism. This effectively balances the contributions of different teacher models during the training process, enhances the adaptability of the loss function to different tasks, and thus improves the performance of the model on various tasks.

[0162] The initial structure and parameters of the CNN-Transformer hybrid architecture are determined according to the input data and task requirements. During the training process, the architecture parameters are dynamically adjusted according to the performance of the model on the validation set, and the reinforcement learning algorithm is used to learn the best adjustment strategy. Beneficial effects: The performance of the model on different tasks and data sets is improved, the limitations of the existing hybrid architecture are overcome, the model can flexibly adapt to various tasks and data changes, and the versatility and generalization ability are enhanced.

[0163] In an embodiment of the present invention, a plurality of visual task teacher models are collected, and the plurality of visual task teacher models are evaluated on a plurality of standard data sets to obtain performance indicators of each visual task teacher model corresponding to different task requirements; and, structural characteristics of each visual task teacher model are determined; generation instructions for a target visual task model are received; target performance indicators corresponding to each visual task teacher model under target task requirements are determined; and, target structural characteristics that meet the characteristics of a target data set are determined; candidate visual task teacher models are determined from a plurality of visual task teacher models based on the target performance indicators and the target structural characteristics; based on an attention mechanism, target attention weights of the candidate visual task teacher models are determined; and a target visual task model is generated based on the target attention weights and the candidate visual task teacher models.

[0164] The embodiment of the present invention breaks through the limitation of relatively fixed selection of teacher models in the existing methods, and proposes a strategy for dynamically selecting teacher models based on task requirements and model performance. According to the characteristics of different visual tasks (such as image classification, semantic segmentation, object detection, etc.), the performance indicators of the teacher model on the corresponding tasks (such as accuracy, recall rate, mIOU, etc.) and the model structure characteristics (such as the ratio of convolutional layers to Transformer layers, model depth, etc.) are comprehensively considered to adaptively select the appropriate teacher model combination.

[0165] In addition, in order to solve the problem that the existing loss function cannot fully balance the contributions of different teachers when dealing with the fusion of multiple teacher models and is not adaptable enough to different tasks, an adaptive loss function is designed. This function can dynamically adjust the weights of feature matching of each teacher model according to the performance of the teacher model in different training stages and different tasks.

[0166] In addition, the existing CNN-Transformer hybrid architecture is relatively fixed in design and cannot be flexibly adjusted according to the characteristics of different tasks and data sets. The present invention proposes a dynamic optimization method for the hybrid architecture. Based on the CNN-Transformer hybrid architecture, the parameters and structures of the CNN layer and the Transformer layer are dynamically adjusted according to the characteristics of the input data (such as image resolution, target size and distribution, etc.) and task requirements (such as requirements for speed and accuracy).

[0167] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0168] Reference Figure 4 , shows a schematic diagram of the structure of a visual model generation system according to an embodiment of the present invention, which may include the following modules:

[0169] The teacher model selection and fusion module 401 is used to receive a generation instruction of a target visual task model, the generation instruction including a target task requirement of the target visual task model and a target data set feature of the target visual task model; determine a candidate visual task teacher model from multiple preset visual task teacher models according to the target task requirement and the target data set feature; determine a target attention weight of the candidate visual task teacher model based on an attention mechanism; and generate a target visual task model according to the target attention weight and the candidate visual task teacher model.

[0170] In one embodiment of the present invention, the system further includes:

[0171] An adaptive loss function module 402 is used to determine a target loss function of a target visual task model, wherein the target loss function includes a basic loss function and a task-specific adjustment item;

[0172] The hybrid architecture dynamic optimization module 403 is used to generate a model architecture of the target visual task model.

[0173] In an embodiment of the present invention, a visual model generation system includes: a teacher model selection and fusion module 401, which is used to receive a generation instruction of a target visual task model, the generation instruction including a target task requirement of the target visual task model and a target data set feature of the target visual task model; according to the target task requirement and the target data set feature, determining a candidate visual task teacher model from a plurality of preset visual task teacher models; based on an attention mechanism, determining a target attention weight of the candidate visual task teacher model; and generating a target visual task model according to the target attention weight and the candidate visual task teacher model.

[0174] The embodiment of the present invention breaks through the limitation of relatively fixed selection of teacher models in the existing methods, and proposes a strategy for dynamically selecting teacher models based on task requirements and model performance. According to the characteristics of different visual tasks (such as image classification, semantic segmentation, object detection, etc.), the performance indicators of the teacher model on the corresponding tasks (such as accuracy, recall rate, mIOU, etc.) and the model structure characteristics (such as the ratio of convolutional layers to Transformer layers, model depth, etc.) are comprehensively considered to adaptively select the appropriate teacher model combination.

[0175] In addition, in order to solve the problem that the existing loss function cannot fully balance the contributions of different teachers when dealing with the fusion of multiple teacher models and is not adaptable to different tasks, an adaptive loss function is designed. This function can dynamically adjust the weight of feature matching of each teacher model according to the performance of the teacher model in different training stages and different tasks.

[0176] In addition, the existing CNN-Transformer hybrid architecture is relatively fixed in design and cannot be flexibly adjusted according to the characteristics of different tasks and data sets. The present invention proposes a dynamic optimization method for the hybrid architecture. Based on the CNN-Transformer hybrid architecture, the parameters and structures of the CNN layer and the Transformer layer are dynamically adjusted according to the characteristics of the input data (such as image resolution, target size and distribution, etc.) and task requirements (such as requirements for speed and accuracy).

[0177] An embodiment of the present invention further provides an electronic device, including a processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein the computer program implements the above-mentioned method for generating a visual model when executed by the processor.

[0178] An embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the method for generating a visual model as described above is implemented.

[0179] As for the device and system embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0180] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0181] Those skilled in the art will appreciate that the embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the embodiments of the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the embodiments of the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0182] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0183] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0184] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0185] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0186] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0187] The above provides a detailed introduction to a method for generating a visual model, a system for generating a visual model, an electronic device and a computer-readable storage medium. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A method for generating a visual model, characterized in that: The method comprises: Receiving a generation instruction of a target visual task model, the generation instruction including a target task requirement of the target visual task model and a target data set feature of the target visual task model; Determining a candidate visual task teacher model from a plurality of preset visual task teacher models according to the target task requirements and the target data set characteristics; Determining a target attention weight of the candidate visual task teacher model based on an attention mechanism; A target visual task model is generated according to the target attention weight and the candidate visual task teacher model.

2. The method according to claim 1, characterized in that The method further comprises: Collect multiple visual task teacher models, and evaluate the multiple visual task teacher models on multiple standard data sets to obtain performance indicators of each visual task teacher model corresponding to different task requirements; and, determine the structural characteristics of the teacher models for each visual task; The step of determining a candidate visual task teacher model from a plurality of preset visual task teacher models according to the target task requirements and the target data set characteristics includes: Determine the target performance indicators corresponding to each visual task teacher model under the target task requirements; and determine the target structural features that meet the characteristics of the target data set; The candidate visual task teacher model is determined from the multiple visual task teacher models according to the target performance indicator and the target structural feature.

3. The method according to claim 1, characterized in that The target loss function of the target visual task model includes a basic loss function and a task-specific adjustment term; The task-specific adjustment term in the target loss function is determined based on the target task requirements.

4. The method according to claim 1, characterized in that The method further comprises: When training the target visual task model, the target attention weight is adjusted based on the performance indicator corresponding to the current target attention weight.

5. The method according to claim 1, characterized in that The method further comprises: When training the target visual task model, the model parameters of the target visual task model are adjusted based on performance indicators corresponding to current model parameters of the target visual task model.

6. The method according to claim 1, characterized in that The method further comprises: When training the target visual task model, the model architecture of the target visual task model is adjusted based on the performance indicator corresponding to the current model architecture of the target visual task model.

7. A system for generating a visual model, characterized in that: The system comprises: The teacher model selection and fusion module is used to receive a generation instruction of a target visual task model, wherein the generation instruction includes a target task requirement of the target visual task model and a target data set feature of the target visual task model; determine a candidate visual task teacher model from a plurality of preset visual task teacher models according to the target task requirement and the target data set feature; determine a target attention weight of the candidate visual task teacher model based on an attention mechanism; and generate a target visual task model according to the target attention weight and the candidate visual task teacher model.

8. The system according to claim 7, characterized in that The system further comprises: An adaptive loss function module, used to determine a target loss function of the target visual task model, wherein the target loss function includes a basic loss function and a task-specific adjustment item; The hybrid architecture dynamic optimization module is used to generate a model architecture of the target visual task model.

9. An electronic device, characterized in that: The method comprises a processor, a memory and a computer program stored in the memory and capable of running on the processor, wherein the computer program, when executed by the processor, implements the method for generating a visual model as claimed in any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method for generating a visual model as claimed in any one of claims 1 to 6 is implemented.

Images