A multi-dimensional evaluation method for visual basic models

Abstract

The present invention discloses a multi-dimensional evaluation method for a visual basic model, which belongs to the field of computer vision. The implementation method of the present invention is as follows: construct a multi-dimensional indicator system for evaluating a visual basic model, including a multi-dimensional visual capability system and a multi-category visual task system. Construct a mapping relationship between the multi-dimensional visual capability system and the multi-category visual task system. Obtain the visual basic model to be evaluated and select the visual capability dimension to be evaluated, and determine the visual task corresponding to the capability dimension. Construct an evaluation network including a visual basic model and a task prediction head for each task, fine-tune the training and test the evaluation network, and obtain the performance index of the current task. Traverse the determined visual tasks, perform evaluation network construction, fine-tune training and performance testing for each visual task, obtain performance indicators of all visual tasks, calculate the visual capability index of the visual basic model according to the obtained performance index, and realize multi-dimensional evaluation of the visual basic model.

Description

Technical Field

[0001] The present invention relates to a multi-dimensional evaluation method for a visual basic model, and belongs to the field of computer vision. Background Art

[0002] With the rapid development of deep learning technology, visual foundation models have become an important tool for solving complex visual tasks. However, the existing visual foundation models have diverse architectures, huge parameters, and difficult-to-control training processes, which poses huge challenges to the exploration of the internal mechanisms of visual foundation models and their industrial implementation. To solve these problems, it is urgent to conduct a comprehensive evaluation of visual foundation models. By evaluating and comparing the performance of various aspects of visual foundation models, it can not only help researchers master the internal mechanisms of visual foundation models and grasp future development directions, but also effectively support the application of visual foundation models in the industry.

[0003] In the era of small models, models are often optimized for specific tasks. The focus of the evaluation is mainly on the performance of a single task and dataset, and this is used as a standard to measure the model's capabilities. However, in the era of large models, since the model has already acquired strong representation capabilities in the pre-training stage, the model can show significant performance advantages in a wide range of downstream tasks through fine-tuning training. It can be seen that the visual basic model has stronger versatility and transfer learning capabilities, making it difficult for a single or partial task evaluation to reflect the versatility and transfer learning capabilities of the visual basic model in detail. Therefore, establishing an evaluation method that can effectively evaluate the multi-dimensional capabilities of the visual basic model has become an urgent need in the field of computer vision. Summary of the invention

[0004] In order to comprehensively evaluate the performance of the visual basic model, the main purpose of the present invention is to provide a multi-dimensional evaluation method for the visual basic model, which can realize the multi-dimensional quantitative evaluation of the visual basic model in a targeted manner. The present invention has the advantages of high evaluation accuracy, high efficiency and good flexibility. The evaluation results according to the present invention can assist the structural analysis and optimization of the visual basic model, select the visual basic model whose visual ability index meets the preset index requirements as the general visual basic model, and solve the related engineering problems in the field of visual recognition, which include scene perception, scene understanding, automatic driving, smart transportation and face recognition.

[0005] The purpose of the present invention is achieved through the following technical solutions.

[0006] The present invention discloses a multi-dimensional evaluation method for a visual basic model, and constructs a multi-dimensional indicator system for evaluating the visual basic model. The multi-dimensional indicator system includes a multi-dimensional visual ability system and a multi-category visual task system. A multi-category visual task integration system is constructed according to the multi-category visual task system. A mapping relationship between the multi-dimensional visual ability system and the multi-category visual task system is constructed. The visual basic model to be evaluated is obtained, and the visual ability dimension to be evaluated is selected, and the visual task corresponding to the visual ability dimension is selected. For each visual task, an evaluation network is constructed using the visual basic model and the task prediction head in the multi-category visual task integration system. The evaluation network is fine-tuned and trained and performance tested, and the performance index of the current visual task is calculated. Each selected visual task is traversed in turn, and the evaluation network construction, evaluation network fine-tuning training and evaluation network performance test are respectively performed for each visual task, and the performance index of all selected visual tasks is obtained. The visual ability index of the visual basic model is calculated according to the performance index of all visual tasks, and the multi-dimensional evaluation of the visual basic model is realized.

[0007] The present invention discloses a multi-dimensional evaluation method for a visual basic model, comprising the following steps:

[0008] Step 1: Based on the characteristics of the visual basic model, a multi-dimensional indicator system for evaluating the visual basic model is constructed; the multi-dimensional indicator system includes two parts: a multi-dimensional visual ability system and a multi-category visual task system;

[0009] The multi-dimensional visual ability system consists of five visual ability dimensions: visual local perception ability dimension, visual global perception ability dimension, visual analogy ability dimension, visual induction ability dimension and visual reasoning ability dimension;

[0010] The multi-category visual task system consists of eight visual tasks: object detection task, semantic segmentation task, instance segmentation task, image classification task, depth estimation task, image retrieval task, semi-supervised image classification task and small sample image classification task;

[0011] According to the multi-category visual task system, a multi-category visual task integration system for multi-dimensional capability evaluation of a visual basic model is constructed; the multi-category visual task integration system includes task prediction heads corresponding to eight visual tasks and performance indicator calculation methods corresponding to eight visual tasks;

[0012] Step 2: Construct a mapping relationship between the multi-dimensional visual ability system described in step 1 and the multi-category visual task system described in step 1; the visual local perception ability dimension corresponds to the object detection task, semantic segmentation task and instance segmentation task; the visual global perception ability dimension corresponds to the image classification task and depth estimation task; the visual analogy ability dimension corresponds to the image retrieval task; the visual induction ability dimension corresponds to the semi-supervised image classification task; the visual reasoning ability dimension corresponds to the small sample image classification task;

[0013] Step 3: Obtain the visual basic model to be evaluated; for the visual basic model to be evaluated, select one or more visual ability dimensions from the five visual ability dimensions in step 1 as the visual ability dimensions to be evaluated; number the selected visual ability dimensions in sequence as ; According to the mapping relationship constructed in step 2, determine the visual task corresponding to the selected visual ability dimension; select the task prediction head corresponding to the selected visual task in the multi-category visual task integration system in step 1; randomly initialize the parameter weights of the selected task prediction head; number the selected task prediction heads in sequence as ;No. The task corresponds to A task prediction head;

[0014] Step 4: Use the visual basic model to be evaluated in step 3 and step 3 The task prediction head constructs an evaluation network for evaluating the visual basic model to be evaluated; obtains the first A visual training dataset for each task; fine-tuning the evaluation network using the visual training dataset to obtain a fine-tuned evaluation network;

[0015] Step 5: Get the A visual evaluation dataset for tasks; The visual evaluation dataset for each task includes images and task annotations. The images of the visual evaluation dataset are input into the visual basic model in the evaluation network fine-tuned in step 4 to obtain the feature map corresponding to the images of the visual evaluation dataset. The feature map is input into the first step of the evaluation network. task prediction head, and get the The task prediction results of the task are compared with the The task prediction results of the task are compared with the The task label of each task is calculated according to the performance index calculation method corresponding to the visual task in step 1. Performance indicators of tasks ;

[0016] Step 6: From 1 to Traverse the task prediction heads selected in step 3 in turn, execute steps 4 and 5 for each task prediction head, and obtain the prediction head determined in step 3. Performance metrics for visual tasks ;

[0017] Step 7: Based on the results obtained in step 6 Performance metrics for visual tasks , combined with the mapping relationship between the multi-dimensional visual ability system constructed in step 2 and the multi-category visual task system, the number of step 3 is determined as The visual ability dimension corresponds to The performance index of the visual task is ; Use the visual ability index calculation method defined in formula (1) to calculate the visual basic model in the number Visual ability indicators of the visual ability dimension ,Will From 1 to Traverse and get Visual ability index , realize multi-dimensional quantitative evaluation of visual basic models;

[0018]

[0019] In formula (1), represents the number of visual tasks; Indicates tasks; Indicates Performance indicators for each task; and Respectively represent The theoretical worst value and theoretical optimal value of the performance index of each task; the depth estimation task Set to 1.

[0020] The visual ability index of the visual basic model obtained in step 7 , realize the structural analysis and optimization of the visual basic model, select the visual basic model whose visual ability indicators meet the preset indicator requirements as the general visual basic model, and solve the relevant engineering problems in the field of visual recognition, which include scene perception, scene understanding, autonomous driving, smart transportation and face recognition.

[0021] The task prediction head of the object detection task uses the Faster R-CNN module; the Faster R-CNN module includes a region generation network submodule and a detector submodule; the region generation network submodule performs a convolution operation on the feature map output by the visual base model to obtain multiple candidate boxes; the detector submodule generates an object detection task prediction result including an object prediction box and a corresponding confidence level based on the multiple candidate boxes and the feature map output by the visual base model; the object detection task prediction result is compared with the task annotation in the visual evaluation dataset of the object detection task, and the performance indicator Box mAP of the object detection task is calculated according to formula (2); the visual local perception ability of the visual base model is evaluated by Box mAP;

[0022] In formula (2), Represents the total number of object categories annotated in the visual evaluation dataset for the object detection task; Indicates object categories; According to the object category The Precision-Recall curve obtained by the corresponding object detection task prediction results and the task annotation of the object detection task; Represents the area below the Precision-Recall curve area.

[0023] The task prediction head of the semantic segmentation task uses the UPerNet module; the UPerNet module includes a feature pyramid submodule and a segmentation prediction submodule; the feature pyramid submodule performs feature enhancement on the feature map output by the visual base model through convolution and upsampling operations, and outputs a feature map whose size is one-fourth of the input image size; the segmentation prediction submodule then upsamples the feature map output by the feature pyramid submodule to obtain the semantic segmentation task prediction result; the semantic segmentation task prediction result is compared with the task annotation in the visual evaluation data set of the semantic segmentation task, and the performance index of the semantic segmentation task is calculated according to formula (3): ;pass Evaluate the local visual perception ability of the visual basic model;

[0024] In formula (3), Represents the total number of semantic categories annotated in the visual evaluation dataset for semantic segmentation tasks; Indicates semantic categories; Represents the semantic category in the prediction result of the semantic segmentation task The corresponding pixel area; Represents the semantic category in the task annotation of the semantic segmentation task The corresponding pixel area; Indicates that the semantic categories in the semantic segmentation task prediction results and the semantic segmentation task task annotations are The pixel area of ​​​​ Indicates that the semantic category in the semantic segmentation task prediction result or the task annotation of the semantic segmentation task is pixel area.

[0025] The task prediction head of the instance segmentation task uses the Mask2former module; the Mask2former module includes a pixel-level submodule, a query submodule and a segmentation submodule; the pixel-level submodule upsamples the feature map output by the visual base model to generate a feature map with the same image size as the visual evaluation dataset of the instance segmentation task; the query submodule calculates the query code to locate the instance object in the feature map output by the visual base model and generates an embedding vector; the segmentation submodule performs a matrix multiplication operation on the feature map output by the pixel-level submodule and the embedding vector output by the query submodule to obtain the instance segmentation task prediction result; the instance segmentation task prediction result is compared with the task annotation in the visual evaluation dataset of the instance segmentation task, and the performance index of the instance segmentation task is calculated according to formula (4): ;pass Evaluate the local visual perception ability of the visual basic model;

[0026]

[0027] In formula (4), Represents the total number of instance categories labeled in the visual evaluation dataset for instance segmentation tasks; Indicates Instance categories; Indicates that based on the instance category The Precision-Recall curve obtained by the corresponding instance segmentation task prediction result and the task annotation of the instance segmentation task; Represents the area below the Precision-Recall curve area.

[0028] The task prediction head for three visual tasks, image classification task, small sample image classification task and semi-supervised image classification task, uses a linear classifier module; the linear classifier module contains a global average pooling layer and a linear layer; the global average pooling layer performs global average pooling on the feature map output by the visual basic model to obtain the pooled feature map; the linear layer performs linear mapping on the pooled feature map to obtain the task prediction result;

[0029] Compare the prediction results of the image classification task with the task annotations in the visual evaluation dataset of the image classification task, and calculate the performance index AccCls of the image classification task according to formula (5); evaluate the visual global perception ability of the visual basic model through the performance index AccCls of the image classification task;

[0030] Compare the prediction results of the small sample image classification task with the task annotations in the visual evaluation dataset of the small sample image classification task, and calculate the performance index AccFew of the small sample image classification task according to formula (6); evaluate the visual reasoning ability of the visual basic model through the performance index AccFew of the small sample image classification task;

[0031] Compare the prediction results of the semi-supervised image classification task with the task annotations in the visual evaluation dataset of the semi-supervised image classification task, and calculate the performance index AccSemi of the semi-supervised image classification task according to formula (7); evaluate the visual induction ability of the visual basic model through the performance index AccSemi of the semi-supervised image classification task;

[0032]

[0033] In formula (5), formula (6) and formula (7), , and They represent the total number of images in the visual evaluation dataset for image classification tasks, the total number of images in the visual evaluation dataset for small sample image classification tasks, and the total number of images in the visual evaluation dataset for semi-supervised image classification tasks respectively; , and They represent the number of images whose task prediction results and task annotations are both positive examples in the image classification task, small sample image classification task, and semi-supervised image classification task respectively; , and They respectively represent the number of images whose task prediction results and task annotations are both negative examples in the image classification task, small sample image classification task, and semi-supervised image classification task.

[0034] The task prediction head of the depth estimation task uses the Adabins module; the Adabins module includes an adaptive calculation submodule and a depth prediction submodule; the adaptive calculation submodule adaptively calculates the width vector for the feature map output by the visual basic model and calculates the range attention map of the image by fusing local features with global feature information; the depth prediction submodule performs matrix multiplication operations on the width vector and the range attention map to obtain the prediction result of the depth estimation task; the prediction result of the depth estimation task is compared with the task annotation in the visual evaluation dataset of the depth estimation task, and the performance indicator RMSE of the depth estimation task is calculated according to formula (8); the visual global perception ability of the visual basic model is evaluated by RMSE;

[0035]

[0036] In formula (8) The total number of pixels in the image representing the visual benchmark dataset for the depth estimation task; Indicates pixels; Indicates the prediction result of the depth estimation task The depth prediction value of pixels; Indicates the task annotation of the depth estimation task. The depth value of the pixel.

[0037] The task prediction head of the image retrieval task uses a linear classifier module; the linear classifier module includes a global average pooling layer and a linear layer; the global average pooling layer performs global average pooling on the feature map output by the visual basic model to obtain the pooled feature map; the linear layer performs linear mapping on the pooled feature map to obtain the image retrieval task prediction result; the image retrieval task prediction result is compared with the task annotation in the visual evaluation data set of the image retrieval task, and the performance indicator Recall of the image retrieval task is calculated according to formula (9); the visual analogy ability of the visual basic model is evaluated by Recall;

[0038] In formula (9), Represents the total number of images in the visual evaluation dataset for the image retrieval task; Represents the first images; Indicates that according to The number of images of the same category correctly retrieved from an image; Indicates that according to The number of images of different categories that were incorrectly retrieved for each image.

[0039] Beneficial Effects

[0040] 1. The present invention discloses a multi-dimensional evaluation method for a visual basic model. According to the characteristics of the visual basic model and the actual application requirements, the visual ability dimension of the visual basic model is divided into the visual local perception ability dimension, the visual global perception ability dimension, the visual analogy ability dimension, the visual induction ability dimension and the visual reasoning ability dimension. The multi-dimensional visual ability system constructed on this basis takes into account both the visual ability coverage and the visual ability evaluation efficiency, and can be used for the multi-dimensional ability efficient evaluation of the visual basic model.

[0041] 2. The present invention discloses a multi-dimensional evaluation method for a visual basic model, constructs a multi-category visual task system consisting of eight visual tasks, namely, object detection task, semantic segmentation task, instance segmentation task, image classification task, depth estimation task, image retrieval task, semi-supervised image classification task and small sample image classification task, and constructs a multi-category visual task integration system for multi-dimensional capability evaluation of the visual basic model; selects task prediction heads in a targeted manner in the multi-category visual task integration system; selects the visual capability dimensions to be evaluated of the visual basic model to be evaluated and the visual tasks corresponding to the visual capability dimensions to be evaluated, and determines the corresponding task prediction heads according to the selected visual tasks; randomly initializes the parameter weights of the determined task prediction heads, and numbers the determined task prediction heads in sequence; combines the obtained visual basic model to be evaluated and the determined first The evaluation network for evaluating the visual basic model to be evaluated is constructed with each task prediction head, and the evaluation network is fine-tuned and trained; the performance test is performed using the fine-tuned evaluation network, and the performance of the visual basic model is evaluated according to the performance indicators of the visual basic model on the selected visual tasks. Traverse step 4 and step 5 in sequence to get Performance indicators for each task , to achieve performance evaluation of the visual basic model.

[0042] 3. The present invention discloses a multi-dimensional evaluation method for a visual basic model, which constructs a mapping relationship between a multi-dimensional visual capability system and a multi-category visual task system. It can select a specific task prediction head according to actual needs to evaluate the performance of the visual basic model in a specified visual capability dimension, thereby improving the flexibility and efficiency of the visual basic model performance evaluation.

[0043] 4. The present invention discloses a multi-dimensional evaluation method for a visual basic model, which combines the mapping relationship between the constructed multi-dimensional visual ability system and the multi-category visual task system to construct a visual ability index for multi-dimensional ability evaluation of the visual basic model. Relationship, based on visual ability index The visual ability index of the visual basic model evaluated by relational calculation under the specified visual ability dimension to be evaluated , realizing multi-dimensional quantitative evaluation of visual basic models.

[0044] 5. The present invention discloses a multi-dimensional evaluation method for a visual basic model. On the basis of achieving the above four beneficial effects, the visual ability index of the visual basic model is used to evaluate the visual ability of the visual basic model. , realize the structural analysis and optimization of the visual basic model, select the visual basic model whose visual ability indicators meet the preset indicator requirements as the general visual basic model, and solve the relevant engineering problems in the field of visual recognition, which include scene perception, scene understanding, autonomous driving, smart transportation and face recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 It is a flow chart of a multi-dimensional evaluation method for a visual basic model disclosed in the present invention;

[0046] Figure 2 It is a structural diagram of a multi-category visual task integration system constructed by the present invention;

[0047] Figure 3 The multi-dimensional capability evaluation results of the visual basic model are shown. DETAILED DESCRIPTION

[0048] In order to better illustrate the purpose and advantages of the present invention, the invention is further described below with reference to the accompanying drawings and examples.

[0049] like Figure 1 As shown, the present embodiment discloses a multi-dimensional evaluation method for a visual basic model, and the specific implementation steps are as follows:

[0050] Step 1: Based on the characteristics of the visual basic model, a multi-dimensional indicator system for evaluating the visual basic model is constructed. The multi-dimensional indicator system includes two parts: a multi-dimensional visual ability system and a multi-category visual task system.

[0051] The multi-dimensional visual ability system consists of five visual ability dimensions: visual local perception ability dimension, visual global perception ability dimension, visual analogy ability dimension, visual induction ability dimension and visual reasoning ability dimension.

[0052] The multi-category visual task system consists of eight visual tasks: object detection task, semantic segmentation task, instance segmentation task, image classification task, depth estimation task, image retrieval task, semi-supervised image classification task and small sample image classification task.

[0053] According to the multi-category visual task system, a multi-category visual task integration system for multi-dimensional capability evaluation of the visual basic model is constructed, such as Figure 2The multi-category visual task integration system comprises task prediction heads corresponding to the eight visual tasks and performance indicator calculation methods corresponding to the eight visual tasks.

[0054] The task prediction head for the object detection task in the multi-category visual task integration system uses the Faster R-CNN module. The Faster R-CNN module includes a region generation network submodule and a detector submodule. The region generation network submodule performs a convolution operation on the feature map output by the visual base model to obtain multiple candidate boxes. The detector submodule generates an object detection task prediction result including an object prediction box and a corresponding confidence level based on the multiple candidate boxes and the feature map output by the visual base model. The object detection task prediction result is compared with the task annotation in the visual evaluation data set of the object detection task, and the performance index of the object detection task is calculated according to formula (10): ,pass Evaluate the visual local perception ability of the visual basic model.

[0055]

[0056] In formula (2), Represents the total number of object categories annotated in the visual evaluation dataset for the object detection task; Indicates object categories; According to the object category The Precision-Recall curve obtained by the corresponding object detection task prediction results and the task annotation of the object detection task; Represents the area below the Precision-Recall curve area.

[0057] The task prediction head of the semantic segmentation task in the multi-category visual task integration system uses the UPerNet module. The UPerNet module includes a feature pyramid submodule and a segmentation prediction submodule. The feature pyramid submodule performs feature enhancement on the feature map output by the visual base model through convolution and upsampling operations, and outputs a feature map whose size is one-fourth of the input image size. The segmentation prediction submodule then upsamples the feature map output by the feature pyramid submodule to obtain the semantic segmentation task prediction result. The semantic segmentation task prediction result is compared with the task annotation in the visual evaluation dataset of the semantic segmentation task, and the performance index of the semantic segmentation task is calculated according to formula (3): ,pass Evaluate the visual local perception ability of the visual basic model.

[0058]

[0059] In formula (3), Represents the total number of semantic categories annotated in the visual evaluation dataset for semantic segmentation tasks; Indicates semantic categories; Represents the semantic category in the prediction result of the semantic segmentation task The corresponding pixel area; Represents the semantic category in the task annotation of the semantic segmentation task The corresponding pixel area; Indicates that the semantic categories in the semantic segmentation task prediction results and the semantic segmentation task task annotations are The pixel area of ​​​​ Indicates that the semantic category in the semantic segmentation task prediction result or the task annotation of the semantic segmentation task is pixel area.

[0060] The task prediction head of the instance segmentation task in the multi-category visual task integration system uses the Mask2former module. The Mask2former module includes a pixel-level submodule, a query submodule and a segmentation submodule. The pixel-level submodule upsamples the feature map output by the visual base model to generate a feature map with the same image size as the visual evaluation data set of the instance segmentation task. The query submodule calculates the query code to locate the instance object in the feature map output by the visual base model and generates an embedding vector. The segmentation submodule performs a matrix multiplication operation on the feature map output by the pixel-level submodule and the embedding vector output by the query submodule to obtain the instance segmentation task prediction result. Compare the instance segmentation task prediction result with the task annotation in the visual evaluation data set of the instance segmentation task, and calculate the performance index of the instance segmentation task according to formula (4): ,pass Evaluate the visual local perception ability of the visual basic model.

[0061]

[0062] In formula (4), Represents the total number of instance categories labeled in the visual evaluation dataset for instance segmentation tasks; Indicates Instance categories; Indicates that based on the instance category The Precision-Recall curve obtained by the corresponding instance segmentation task prediction result and the task annotation of the instance segmentation task; Represents the area under the Precision-Recall curve area.

[0063] The task prediction head of the three visual tasks of image classification task, small sample image classification task and semi-supervised image classification task in the multi-category visual task integration system uses a linear classifier module. The linear classifier module includes a global average pooling layer and a linear layer. The global average pooling layer performs global average pooling on the feature map output by the visual basic model to obtain the pooled feature map. The linear layer performs linear mapping on the pooled feature map to obtain the task prediction result. The image classification task prediction result is compared with the task annotation in the visual evaluation data set of the image classification task, and the performance index AccCls of the image classification task is calculated according to formula (5). The visual global perception ability of the visual basic model is evaluated by the performance index AccCls of the image classification task. The small sample image classification task prediction result is compared with the task annotation in the visual evaluation data set of the small sample image classification task, and the performance index AccFew of the small sample image classification task is calculated according to formula (6). The visual reasoning ability of the visual basic model is evaluated by the performance index AccFew of the small sample image classification task. Compare the prediction results of the semi-supervised image classification task with the task annotations in the visual evaluation dataset of the semi-supervised image classification task, and calculate the performance index AccSemi of the semi-supervised image classification task according to formula (7). The visual induction ability of the visual basic model is evaluated by the performance index AccSemi of the semi-supervised image classification task.

[0064]

[0065] In formula (5), formula (6) and formula (7), , and They represent the total number of images in the visual evaluation dataset for image classification tasks, the total number of images in the visual evaluation dataset for small sample image classification tasks, and the total number of images in the visual evaluation dataset for semi-supervised image classification tasks respectively; , and They represent the number of images whose task prediction results and task annotations are both positive examples in the image classification task, small sample image classification task, and semi-supervised image classification task respectively; , and They respectively represent the number of images whose task prediction results and task annotations are both negative examples in the image classification task, small sample image classification task, and semi-supervised image classification task.

[0066] The task prediction head of the depth estimation task in the multi-category visual task integration system uses the Adabins module. The Adabins module includes an adaptive calculation submodule and a depth prediction submodule. The adaptive calculation submodule adaptively calculates the width vector for the feature map output by the visual basic model and calculates the range attention map of the image by fusing local features with global feature information. The depth prediction submodule performs matrix multiplication operations on the width vector and the range attention map to obtain the prediction result of the depth estimation task. The prediction result of the depth estimation task is compared with the task annotation in the visual evaluation data set of the depth estimation task, and the performance index of the depth estimation task is calculated according to formula (8): ,pass Evaluate the global visual perception ability of the visual basic model.

[0067]

[0068] In formula (8), The total number of pixels in the image representing the visual benchmark dataset for the depth estimation task; Indicates pixels; Indicates the prediction result of the depth estimation task The depth prediction value of pixels; Indicates the task annotation of the depth estimation task. The depth value of the pixel.

[0069] The task prediction head of the image retrieval task in the multi-category visual task integration system uses a linear classifier module. The linear classifier module includes a global average pooling layer and a linear layer. The global average pooling layer performs global average pooling on the feature map output by the visual basic model to obtain a pooled feature map. The linear layer performs linear mapping on the pooled feature map to obtain the image retrieval task prediction result. The image retrieval task prediction result is compared with the task annotation in the visual evaluation data set of the image retrieval task, and the performance index of the image retrieval task is calculated according to formula (9): ;pass Evaluating the visual analogy capabilities of vision-based models.

[0070]

[0071] In formula (9), Represents the total number of images in the visual evaluation dataset for the image retrieval task; Represents the first images; Indicates that according to The number of images of the same category correctly retrieved from an image; Indicates that according to The number of images of different categories that were incorrectly retrieved for each image.

[0072] Step 2: Construct a mapping relationship between the multi-dimensional visual ability system described in step 1 and the multi-category visual task system described in step 1. The visual local perception ability dimension corresponds to the object detection task, semantic segmentation task, and instance segmentation task. The visual global perception ability dimension corresponds to the image classification task and depth estimation task. The visual analogy ability dimension corresponds to the image retrieval task. The visual induction ability dimension corresponds to the semi-supervised image classification task. The visual reasoning ability dimension corresponds to the small sample image classification task.

[0073] Step 3: Obtain the visual basic model EVA-L to be evaluated. For the visual basic model EVA-L to be evaluated, select the visual local perception ability dimension, visual global perception ability dimension, visual analogy ability dimension, visual induction ability dimension and visual reasoning ability dimension from the five visual ability dimensions described in step 1 as the visual ability dimensions to be evaluated. The selected visual ability dimensions are numbered in sequence as ; According to the mapping relationship between the multi-dimensional visual ability system and the multi-category visual task system constructed in step 2, determine the visual tasks corresponding to the five selected visual ability dimensions, including object detection tasks, semantic segmentation tasks, instance segmentation tasks, image classification tasks, depth estimation tasks, image retrieval tasks, semi-supervised image classification tasks and small sample image classification tasks. Select the task prediction heads corresponding to the eight selected visual tasks in the multi-category visual task integration system described in step 1. The task prediction head corresponding to the object detection task is the Faster R-CNN module; the task prediction head corresponding to the semantic segmentation task is the UPerNet module; the task prediction head corresponding to the instance segmentation task is the Mask2former module; the task prediction head corresponding to the image classification task is the linear classifier module; the task prediction head corresponding to the depth estimation task is the Adabins module; the task prediction head corresponding to the image retrieval task is the linear classifier module; the task prediction head corresponding to the semi-supervised image classification task is the linear classifier module; and the task prediction head corresponding to the small sample image classification task is the linear classifier module. Randomly initialize the parameter weights of the eight selected task prediction heads. The selected task prediction heads are numbered in sequence as ; Definition The task corresponds to A task prediction head.

[0074] Step 4: Use the visual basic model to be evaluated in step 3 and the The task prediction head constructs an evaluation network for evaluating the visual basic model to be evaluated. The visual training data set is used to fine-tune the evaluation network to obtain a fine-tuned evaluation network.

[0075] Step 5: Get the A visual evaluation dataset for tasks. The visual evaluation dataset for each task includes images and task annotations. The images of the visual evaluation dataset are input into the visual basic model in the evaluation network fine-tuned in step 4 to obtain the feature map corresponding to the images of the visual evaluation dataset. The feature map is input into the first step of the evaluation network. task prediction head, and get the The task prediction results of the task are compared with the The task prediction results of the task are compared with the The task label of each task is calculated according to the performance indicator calculation method corresponding to the visual task in step 1. Performance indicators for each task .

[0076] Step 6: From arrive Traverse the task prediction heads selected in step 3 in sequence, execute steps 4 and 5 for each task prediction head, and obtain the task prediction heads determined in step 3. Performance metrics for visual tasks .

[0077] The visual basic model for evaluation obtained in step 6 is The performance indicators of a visual task are used as auxiliary information, and a visual basic model whose performance indicators meet the preset requirements is selected. The accuracy and efficiency of visual recognition are improved according to the selected visual basic model, so as to solve the relevant engineering problems in the field of visual recognition, which include scene perception, scene understanding, autonomous driving, smart transportation and face recognition.

[0078] Step 7: Based on the performance indicators of the 8 visual tasks obtained in step 6 , combined with the mapping relationship between the multi-dimensional visual ability system constructed in step 2 and the multi-category visual task system, the number of step 3 is determined as The visual ability dimension corresponds to The performance index of the visual task is ;Will From 1 to Traverse, and use the visual ability index calculation method defined in formula (1) to calculate the visual basic model in the numbered Visual ability indicators of the visual ability dimension ,get Visual ability index , realizing multi-dimensional quantitative evaluation of visual basic models.

[0079]

[0080] In formula (1), represents the number of visual tasks; Indicates tasks; Indicates Performance indicators for each task; and Respectively represent The theoretical worst value and theoretical optimal value of the performance index of each task; the depth estimation task Set to 1.

[0081] According to the visual basic model obtained in step 7 Visual ability index , realize the structural analysis and optimization of the visual basic model, select the visual basic model whose visual ability indicators meet the preset indicator requirements as the general visual basic model, and solve the relevant engineering problems in the field of visual recognition, which include scene perception, scene understanding, autonomous driving, smart transportation and face recognition.

[0082] This example will illustrate the effectiveness of this method through evaluation indicators.

[0083] 1. Experimental conditions

[0084] The software configuration of this experiment is Ubuntu 20.04.2 LTS operating system, PyTorch deep learning development framework, Python development language, and the hardware uses Intel(R) Xeon(R) Platinum 8358 CPU @ 2.60GHz and NVIDIAA800 GPU.

[0085] 2. Experimental results

[0086] In order to illustrate the effectiveness of the present invention, the present invention evaluates four mainstream visual basic models, namely, EVA-L, ConvNeXt-v2-L, Swin-L and DINO-v2-G, from five visual ability dimensions: visual local perception ability, visual global perception ability, visual analogy ability, visual induction ability and visual reasoning ability. The evaluation results are as follows: Figure 3As shown in the figure, the performance of the DINO-v2-G model in the dimension of local visual perception is significantly better than that of the other three basic visual models. The performance of the ConvNeXt-v2-L model in the dimensions of global visual perception, visual induction and visual reasoning is significantly superior. The performance of the Swin-L model in the dimension of visual analogy is significantly superior. The performance of the EVA-L model in the five dimensions of visual ability is relatively balanced, and it performs well in the dimensions of global visual perception, visual induction and visual reasoning.

[0087] The specific description above has detailed the purpose, design scheme and beneficial effects of the invention. It should be understood that the above is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A multi-dimensional evaluation method for a visual basic model, characterized by: The method comprises the following steps: step 1: constructing a multidimensional indicator system for evaluating the visual basic model according to the characteristics of the visual basic model; the multidimensional indicator system comprises two parts: a multidimensional visual ability system and a multi-category visual task system; the multidimensional visual ability system consists of five visual ability dimensions, namely, visual local perception ability dimension, visual global perception ability dimension, visual analogy ability dimension, visual induction ability dimension and visual reasoning ability dimension; the multi-category visual task system consists of eight visual tasks, namely, object detection task, semantic segmentation task, instance segmentation task, image classification task, depth estimation task, image retrieval task, semi-supervised image classification task and small sample image classification task; constructing a multi-category visual task integration system for multi-dimensional ability evaluation of the visual basic model according to the multi-category visual task system; the multi-category visual task The integrated system includes task prediction heads corresponding to eight visual tasks and performance indicator calculation methods corresponding to the eight visual tasks; Step 2: Construct a mapping relationship between the multi-dimensional visual ability system described in Step 1 and the multi-category visual task system described in Step 1; the visual local perception ability dimension corresponds to the object detection task, the semantic segmentation task and the instance segmentation task; the visual global perception ability dimension corresponds to the image classification task and the depth estimation task; the visual analogy ability dimension corresponds to the image retrieval task; the visual induction ability dimension corresponds to the semi-supervised image classification task; the visual reasoning ability dimension corresponds to the small sample image classification task; Step 3: Obtain the visual basic model to be evaluated; For the visual basic model to be evaluated, select one or more visual ability dimensions from the five visual ability dimensions in Step 1 as the visual ability dimensions to be evaluated; and number the selected visual ability dimensions in sequence as ; According to the mapping relationship constructed in step 2, determine the visual task corresponding to the selected visual ability dimension; select the task prediction head corresponding to the selected visual task in the multi-category visual task integration system in step 1; randomly initialize the parameter weights of the selected task prediction head; number the selected task prediction heads in sequence as ;No. The task corresponds to task prediction head; Step 4: Use the visual basic model to be evaluated in Step 3 and the The task prediction head constructs an evaluation network for evaluating the visual basic model to be evaluated; obtains the first Visual training datasets for tasks; Use the visual training data set to fine-tune the evaluation network to obtain the fine-tuned evaluation network; Step 5: Get the A visual evaluation dataset for tasks; The visual evaluation dataset for each task includes images and task annotations; Input the image of the visual evaluation dataset into the visual basic model in the evaluation network fine-tuned in step 4 to obtain the feature map corresponding to the image of the visual evaluation dataset; input the feature map into the first task prediction head, and get the The task prediction results of the task are compared with the The task prediction results of the task are compared with the The task label of each task is calculated according to the performance indicator calculation method corresponding to the visual task in step 1. Performance indicators of tasks ; Step 6: From 1 to Traverse the task prediction heads selected in step 3 in turn, execute steps 4 and 5 for each task prediction head, and obtain the prediction head determined in step 3. Performance metrics for visual tasks ; Step 7: Based on the results obtained in step 6 Performance metrics for visual tasks , combined with the mapping relationship between the multi-dimensional visual ability system constructed in step 2 and the multi-category visual task system, the number of step 3 is determined as The visual ability dimension corresponds to The performance index of the visual task is ; The visual ability index calculation method defined in formula (1) is used to calculate the visual basic model in the number Visual ability indicators of the visual ability dimension ,Will From 1 to Traverse and get Visual ability index , realize multi-dimensional quantitative evaluation of visual basic models; In formula (1), represents the number of visual tasks; Indicates tasks; Indicates Performance indicators for each task; and Respectively represent The theoretical worst value and theoretical optimal value of the performance index of each task; the depth estimation task Set to 1.

2. A multi-dimensional evaluation method for a visual basic model according to claim 1, characterized in that: The task prediction head of the object detection task uses the Faster R-CNN module; the Faster R-CNN module includes a region generation network submodule and a detector submodule; the region generation network submodule performs a convolution operation on the feature map output by the visual basic model to obtain multiple candidate boxes; The detector submodule generates an object detection task prediction result including an object prediction box and a corresponding confidence level based on multiple candidate boxes and feature maps output by the visual basic model. The object detection task prediction result is compared with the task annotation in the visual evaluation dataset of the object detection task, and the performance indicator Box mAP of the object detection task is calculated according to formula (2). The visual local perception ability of the visual basic model is evaluated by Box mAP. In formula (2), Represents the total number of object categories annotated in the visual evaluation dataset for the object detection task; Indicates object categories; According to the object category The Precision-Recall curve obtained by the corresponding object detection task prediction results and the task annotation of the object detection task; Represents the area under the Precision-Recall curve area.

3. A multi-dimensional evaluation method for a visual basic model according to claim 1, characterized in that: The task prediction head of the semantic segmentation task uses the UPerNet module; the UPerNet module includes a feature pyramid submodule and a segmentation prediction submodule; the feature pyramid submodule performs feature enhancement on the feature map output by the visual base model through convolution and upsampling operations, and outputs a feature map whose size is one-fourth of the input image size; the segmentation prediction submodule then upsamples the feature map output by the feature pyramid submodule to obtain the semantic segmentation task prediction result; the semantic segmentation task prediction result is compared with the task annotation in the visual evaluation data set of the semantic segmentation task, and the performance index of the semantic segmentation task is calculated according to formula (3): ;pass Evaluate the local visual perception ability of the visual basic model; In formula (3), Represents the total number of semantic categories annotated in the visual evaluation dataset for semantic segmentation tasks; Indicates semantic categories; Represents the semantic category in the prediction result of the semantic segmentation task The corresponding pixel area; Represents the semantic category in the task annotation of the semantic segmentation task The corresponding pixel area; Indicates that the semantic categories in the semantic segmentation task prediction results and the semantic segmentation task task annotations are The pixel area of ​​​​ Indicates that the semantic category in the semantic segmentation task prediction result or the task annotation of the semantic segmentation task is pixel area.

4. A multi-dimensional evaluation method for a visual basic model as claimed in claim 1, characterized in that: The task prediction head of the instance segmentation task uses the Mask2former module; the Mask2former module includes a pixel-level submodule, a query submodule, and a segmentation submodule; the pixel-level submodule upsamples the feature map output by the visual base model to generate a feature map with the same image size as the visual evaluation dataset of the instance segmentation task; The query submodule calculates the query encoding to locate the instance object in the feature map output by the visual base model and generates an embedding vector; The segmentation submodule performs matrix multiplication operations on the feature map output by the pixel-level submodule and the embedding vector output by the query submodule to obtain the prediction result of the instance segmentation task; the prediction result of the instance segmentation task is compared with the task annotation in the visual evaluation dataset of the instance segmentation task, and the performance indicator Mask mAP of the instance segmentation task is calculated according to formula (4); the visual local perception ability of the visual basic model is evaluated by Mask mAP; In formula (4), Represents the total number of instance categories labeled in the visual evaluation dataset for instance segmentation tasks; Indicates Instance categories; Indicates that based on the instance category The Precision-Recall curve obtained by the corresponding instance segmentation task prediction result and the task annotation of the instance segmentation task; Represents the area under the Precision-Recall curve area.

5. The multi-dimensional evaluation method for a visual basic model according to claim 1, characterized in that: The task prediction head of three visual tasks, image classification task, small sample image classification task and semi-supervised image classification task, uses a linear classifier module; the linear classifier module includes a global average pooling layer and a linear layer; the global average pooling layer performs global average pooling on the feature map output by the visual basic model to obtain a pooled feature map; The linear layer performs linear mapping on the pooled feature map to obtain the task prediction result; Compare the prediction results of the image classification task with the task annotations in the visual evaluation dataset of the image classification task, and calculate the performance index AccCls of the image classification task according to formula (5); evaluate the visual global perception ability of the visual basic model through the performance index AccCls of the image classification task; compare the prediction results of the small sample image classification task with the task annotations in the visual evaluation dataset of the small sample image classification task, and calculate the performance index AccFew of the small sample image classification task according to formula (6); evaluate the visual reasoning ability of the visual basic model through the performance index AccFew of the small sample image classification task; compare the prediction results of the semi-supervised image classification task with the task annotations in the visual evaluation dataset of the semi-supervised image classification task, and calculate the performance index AccSemi of the semi-supervised image classification task according to formula (7); evaluate the visual induction ability of the visual basic model through the performance index AccSemi of the semi-supervised image classification task; In formula (5), formula (6) and formula (7), , and They represent the total number of images in the visual evaluation dataset for image classification tasks, the total number of images in the visual evaluation dataset for small sample image classification tasks, and the total number of images in the visual evaluation dataset for semi-supervised image classification tasks respectively; , and They represent the number of images whose task prediction results and task annotations are both positive examples in the image classification task, small sample image classification task, and semi-supervised image classification task respectively; , and They respectively represent the number of images whose task prediction results and task annotations are both negative examples in the image classification task, small sample image classification task, and semi-supervised image classification task.

6. A multi-dimensional evaluation method for a visual basic model according to claim 1, characterized in that: The task prediction head of the depth estimation task uses the Adabins module; the Adabins module includes an adaptive calculation submodule and a depth prediction submodule; the adaptive calculation submodule adaptively calculates the width vector for the feature map output by the visual basic model and calculates the range attention map of the image by fusing local features with global feature information; the depth prediction submodule performs matrix multiplication operations on the width vector and the range attention map to obtain the prediction result of the depth estimation task; the prediction result of the depth estimation task is compared with the task annotation in the visual evaluation dataset of the depth estimation task, and the performance indicator RMSE of the depth estimation task is calculated according to formula (8); The global visual perception ability of the visual basic model is evaluated by RMSE; In formula (8) The total number of pixels in the image representing the visual benchmark dataset for the depth estimation task; Indicates pixels; Indicates the prediction result of the depth estimation task The depth prediction value of pixels; Indicates the task annotation of the depth estimation task. The depth value of the pixel.

7. A multi-dimensional evaluation method for a visual basic model according to claim 1, characterized in that: The task prediction head of the image retrieval task uses a linear classifier module; The linear classifier module includes a global average pooling layer and a linear layer; the global average pooling layer performs global average pooling on the feature map output by the visual basic model to obtain a pooled feature map; the linear layer performs linear mapping on the pooled feature map to obtain an image retrieval task prediction result; Compare the prediction results of the image retrieval task with the task annotations in the visual evaluation dataset of the image retrieval task, and calculate the performance index Recall of the image retrieval task according to formula (9); Recall is used to evaluate the visual analogy ability of the visual basic model; In formula (9), Represents the total number of images in the visual evaluation dataset for the image retrieval task; Represents the first images; Indicates that according to The number of images of the same category correctly retrieved from an image; Indicates that according to The number of images of different categories that were incorrectly retrieved for each image.

Images