A multi-task visual perception and content processing evaluation method and system for complex scenes

Abstract

The application relates to the technical field of artificial intelligence, and discloses a multi-task visual perception and content processing evaluation method and system for complex scenes. The system adopts a four-layer reference architecture including an infrastructure layer, a data layer, an algorithm model layer and an application layer, covers two dimensions of visual understanding and visual generation, and supports 32 visual tasks such as image classification, target detection and semantic segmentation. The evaluation method integrates 12 performance indicators through a test platform, combines a standardized subjective evaluation mechanism, realizes comprehensive quantitative evaluation of the performance of a visual large model system, and provides a complete technical solution for the standardized design and test of the visual large model system.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a multi-task visual perception and content processing evaluation method and system for complex scenarios. Background Technology

[0002] With the continuous evolution of deep learning technology, large-scale computer vision models, thanks to their powerful feature extraction and task generalization capabilities, have made breakthroughs in multiple tasks such as image classification, object detection, semantic segmentation, and visual content generation, and are gradually extending to practical applications such as industrial quality inspection, intelligent transportation, and remote sensing. However, most existing vision systems are designed for specific tasks or limited task scenarios, with visual understanding and visual generation capabilities separated, making it difficult to support concurrent processing of multiple visual needs under a unified architecture. Faced with the coexistence of multiple visual tasks in complex application scenarios, existing solutions typically require the deployment of multiple independent specialized models, leading to problems such as complex overall architecture, low utilization of computing resources, and insufficient cross-task knowledge sharing capabilities, severely restricting the large-scale application of large-scale vision model systems in engineering practice.

[0003] In terms of system evaluation, existing evaluation systems suffer from a dual dilemma: limited coverage and a lack of standards. The evaluation of visual understanding and visual generation capabilities is independent, lacking a unified benchmark covering the entire task chain. Existing evaluation metrics primarily rely on single objective measures, which are insufficient to accurately reflect visual quality and content rationality for content creation tasks such as image completion, style transfer, and video generation. Furthermore, the lack of standardized subjective evaluation processes and evaluation organizations significantly reduces the credibility of cross-system comparisons. These shortcomings severely hinder the standardized testing and engineering selection of large-scale visual model systems, necessitating the development of a unified technical solution that integrates multi-task visual processing and a systematic evaluation mechanism. Summary of the Invention

[0004] (1) Technical problems to be solved The purpose of this invention is to provide a multi-task visual perception and content processing evaluation method and system for complex scenarios, in order to solve the shortcomings of existing vision systems in terms of unified multi-task processing and systematic evaluation: the visual understanding and visual generation capabilities of existing systems are separated, and there is a lack of a unified architecture to support the concurrent processing of 32 visual tasks; the existing evaluation system has a separation between subjective and objective evaluation, incomplete task coverage, and lacks a standardized comprehensive evaluation method.

[0005] (2) Technical solution To achieve the above objectives, on the one hand, the present invention provides a multi-task visual perception and content processing evaluation method for complex scenarios, including: Step S1: Build a multi-task visual perception and content processing system. The system is based on a four-layer reference architecture that includes an infrastructure layer, a data layer, an algorithm model layer, and an application layer. The system supports the processing of two major categories of tasks: visual understanding and visual generation through a large computer vision model in the algorithm model layer.

[0006] Step S2: Based on the visual processing capabilities declared by the system under test, determine the corresponding evaluation tasks from the preset visual understanding task set and visual generation task set, and equip each evaluation task with a corresponding test dataset.

[0007] Step S3: Send a test request to the system under test through the test platform and obtain the processing results of the test sample by the system under test.

[0008] Step S4: For each evaluation task, select the corresponding objective evaluation indicators according to the task type, quantify the processing results, and obtain the values ​​of each objective performance indicator.

[0009] Step S5: For evaluation tasks that require subjective judgment, an evaluation group is formed according to the prescribed subjective evaluation process to subjectively score the output results of the tested system and obtain a subjective evaluation score.

[0010] Step S6: Combine the objective performance index values ​​with the subjective evaluation scores to obtain the comprehensive performance evaluation results of the tested system on each evaluation task.

[0011] Furthermore, the visual understanding task set includes image classification, video classification, object detection, anomaly detection, rotation detection, change detection, distance measurement, velocity measurement, object counting, object recognition, optical character recognition, face recognition, content feature recognition, pose recognition, event or behavior recognition, relationship recognition, image description, image and text understanding, visual content multi-turn dialogue, mathematical calculation, and table understanding; the visual generation task set includes instance segmentation, interactive segmentation, image missing completion, video completion, super-resolution, semantic extension, object replacement, style transfer, image stitching, fusion, and removal.

[0012] Furthermore, in step S2, the number of test dataset samples for each evaluation task is not less than 2000, and no less than 200 samples are randomly sampled from the corresponding dataset during a single test; when multiple systems to be tested are tested in the same batch, the same test set, sampling method and evaluation method are used.

[0013] Furthermore, in step S4, the objective evaluation metrics include TOP1 accuracy, TOP5 accuracy, intersection-over-union ratio, recognition rate, false recognition rate, missed recognition rate, mean absolute percentage error, average precision, keypoint similarity, average pixel accuracy, image similarity, and sliding window dialogue relevance score, totaling 12 items. The tasks and calculation methods corresponding to each metric are as follows: The formulas for calculating TOP1 accuracy and TOP5 accuracy are as follows: ; ; The intersection-over-union ratio (IoU) is calculated using the following formula: ; In the formula, B1 is the intersection-union ratio; B2 is the labeled bounding box; B3 is the predicted bounding box. In detection and recognition scenarios with temporal continuity, such as events and actions, B1 and B2 respectively indicate the temporal boundary range of the labeled and predicted bounding boxes. In this case, the indicator is called the "temporal intersection-union ratio".

[0014] The recognition rate R, false recognition rate FAR, and false rejection rate FRR are defined as follows: ; ; ; In facial recognition, the numerator of the false recognition rate is the number of samples where two images of different people are incorrectly identified as belonging to the same person, while the numerator of the false recognition rate is the number of samples where the system fails to recognize the target face. In optical character recognition, the numerator of the false recognition rate is the number of samples where a specific character is identified as an incorrect character, while the numerator of the false recognition rate is the number of samples where a specific character is not recognized.

[0015] Mean absolute percentage error (MAPE) is defined as: ; In the formula, MAPE is the mean absolute percentage error; It is the first The true value of each sample; It is the first Predicted values ​​for each sample; It represents the total number of samples.

[0016] Mean accuracy (mAP) is defined as: ; In the formula, Average precision; Total number of categories; For the first kind; For the first The number of predicted classes; For the first The number of correct predictions for a class.

[0017] Keypoint similarity (OKS) is defined as: ; In the formula, OKS is the similarity of key points of objects; It is the identifier of a person or object in the truth value; It is a key marker of a person or object; Indicate key points It has visibility (i.e., it is visible in an image or video); Represents the square root of the area occupied by a person or object (calculated based on the annotation box in the truth value); Indicates the first The normalization factor for each key point (generally used as a constant, obtained by manually annotating all true key points in the sample set with the standard deviation of the true values).

[0018] The above formula calculates the keypoint similarity between two people or two objects. If M people or objects appear in a given image, and N people or objects are predicted, then... and Key point similarity calculation The matrix is ​​then used to select the value with the highest similarity as the predicted value.

[0019] Average pixel accuracy (PA) is defined as: ; In the formula, It is the average pixel accuracy. It is the first One sample; It represents the total number of samples.

[0020] Image similarity is assessed using the Structural Similarity Scale (SSIM) method, defined as: ; In the formula, Image after stitching Image with true value Image similarity; or Representing an image or The signal mean; or Representing an image or The standard deviation of the signal; Represents the covariance of the signal; and It is a constant.

[0021] Sliding window dialogue relevance score ( ) is defined as: ; In the formula, w is the size of the sliding window; R is the number of dialogue turns (the number of times the model responds). This is the r-th response from the model in one dialogue session. This response needs to be manually verified for relevance within a sliding window w. If the number of dialogue rounds in the above example is less than w, then that number of rounds will be used as the window.

[0022] Furthermore, in step S5, the subjective evaluation is implemented by the evaluation group according to the following process: a subset of test samples is randomly and proportionally allocated to each member of the evaluation group, and each member independently completes the evaluation for a single indicator; for each test sample, the input, output, and reference value (if present) are combined into a tuple, which is then presented to the evaluators for a specified duration p; the evaluators provide a score (e.g., on a 5-point scale) within the scoring period g; after scoring, the scored tuple is removed, and the next sample is presented after a time interval d; for evaluators whose overall evaluation rate is less than the specified proportion r, their evaluation results are not included in the total score, and replacements are drawn from the reserve group; a single-round or multi-round same subset scoring method is adopted, and the average score of multiple rounds is taken when the same subset is used; the subjective average total score G is calculated by averaging all valid scores. ; In the formula, The subjective average total score, For the first The score given by each evaluator (e.g., on a 5-point scale, natural numbers 1-5). This represents the number of subjective ratings.

[0023] The evaluation group consists of The team was composed of randomly selected personnel divided into an implementation group and a reserve group, each containing [number of members]. People people( ); Number of implementation team members The number of evaluators should be equal to the number of test sample subsets; evaluators should possess the basic abilities required for evaluation (such as reading specific languages ​​and texts, visual accessibility, etc.), and should preferably possess basic knowledge of large visual model systems (such as clearly understanding the input and output patterns of the large model being evaluated).

[0024] On the other hand, this invention also provides a multi-task visual perception and content processing system for complex scenarios. The system comprises an infrastructure layer, a data layer, an algorithm model layer, and an application layer. The infrastructure layer includes computing devices, storage devices, network transmission devices, and visual signal acquisition and output devices, providing virtualization and task scheduling support for the system. The data layer includes various visual data such as images (including 3D images), videos (including 3D videos), image-text pairs, video-text pairs, and visual sensor signals, supporting visual data retrieval, synthesis, annotation, and quality assessment functions. The algorithm model layer, centered on a large computer vision model, includes pre-processing, post-processing, and model data encryption / decryption components, supporting model compression and conversion, reinforcement learning, fine-tuning, and pre-training model optimization functions. The application layer supports workflow orchestration, pre-configured template configuration, and visual model performance evaluation, providing task management and execution support for business scenarios.

[0025] (3) Beneficial effects Compared with existing technologies, the multi-task visual perception and content processing system provided by this invention is based on a four-layer reference architecture, which integrates the two major capability dimensions of visual understanding and visual generation into a unified framework, supports the concurrent processing of 32 visual tasks, and effectively solves the problems of task fragmentation and complex architecture in existing systems. The supporting evaluation method integrates 12 quantitative performance indicators and a standardized subjective evaluation process, covering both objective accuracy measurement and subjective quality perception, realizing a comprehensive evaluation of the entire visual large model system, and providing systematic technical support for the design optimization and standardized testing of the visual large model system. Attached Figure Description

[0026] To more clearly illustrate the technical solutions of the exemplary embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly described below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1 This is a reference architecture diagram of the multi-task visual perception and content processing system for complex scenarios according to the present invention; Figure 2 This is a schematic diagram of the overall process of the evaluation method of the present invention; Figure 3 This is a schematic diagram of the subjective evaluation process of the present invention. Detailed Implementation

[0028] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0029] Throughout this specification, references to "an embodiment," "an example," or "an example" mean that a particular feature, structure, or characteristic described in connection with that embodiment or example is included in at least one embodiment of the invention. Therefore, the phrases "an embodiment," "an example," "an example," or "an example" appearing in various places throughout the specification do not necessarily refer to the same embodiment or example. Furthermore, specific features, structures, or characteristics can be combined in one or more embodiments or examples in any suitable combination and / or sub-combination. Moreover, those skilled in the art will understand that the illustrations provided herein are for illustrative purposes and are not necessarily drawn to scale. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0030] Example: Figure 1 As shown, this embodiment provides a multi-task visual perception and content processing system for complex scenarios. The system is built on a four-layer reference architecture, including an infrastructure layer, a data layer, an algorithm model layer, and an application layer.

[0031] The infrastructure layer includes computing devices, storage devices, network transmission devices, and visual signal acquisition and output devices, providing unified computing resource management for upper layers through virtualization and task scheduling. The data layer contains various forms of visual data, including images (including 3D images), videos (including 3D videos), image-text pairs, video-text pairs, and visual sensor signals. It provides visual data retrieval, synthesis, annotation, and quality assessment functions to support the training and inference of upper-layer models. The algorithm model layer is centered on a large computer vision model, including pre-processing, post-processing, and model data encryption / decryption components. It supports model compression and conversion, reinforcement learning, fine-tuning, and pre-training for model optimization. The application layer supports workflow orchestration, pre-configured template configuration, and visual model performance evaluation, providing task management and execution support for specific business scenarios.

[0032] The system supports two main categories of tasks: visual understanding and visual generation. Visual understanding capabilities encompass five sub-capabilities: classification, detection, measurement, recognition, and visual representation.

[0033] The classification sub-capabilities include image classification and video classification. For image classification, the system can output one or more predefined categories to which an image or a specified object in an image belongs. It should be able to perform classification under different viewing angles (e.g., front, back, top, bottom, left, right, or other angles), different sizes, different lighting conditions (e.g., brightness, saturation), object deformation, background interference, occlusion, and different resolutions. Classification performance is measured by TOP1 accuracy or TOP5 accuracy, and processing efficiency is measured by end-to-end inference latency or actual throughput. For video classification, the system can output one or more predefined categories to which a video or a specified object in a video belongs. It should be able to classify videos with a playback duration of at least 60 seconds (30FPS).

[0034] The detection sub-capabilities include object detection, anomaly detection, rotation detection, change detection, and object tracking. The object detection system should be able to identify pre-defined targets in an image and output bounding boxes. It should ideally be able to detect small targets (e.g., 100px × 100px) and support custom labels (e.g., natural language descriptions). Detection performance should be measured by intersection-over-union (IoU) or average precision. The anomaly detection system should be able to detect images containing anomalous objects or phenomena. It should ideally provide bounding boxes for anomalous regions. Detection performance should be measured by false negative and false positive rates, and anomaly localization performance should be measured by IoU. The rotation detection system should be able to identify pre-defined objects in an image and output rotated bounding boxes. Performance should be measured by polygon intersection-over-union. The change detection system should be able to detect changes in specific objects in an image or video. It should ideally be able to identify the type of change (e.g., disappearance, appearance, color change, or deformation). Performance should be measured by recognition rate, false recognition rate, or false negative rate. The target tracking system should be able to detect and mark the target to be tracked in video with a frame rate of not less than 25 frames / s, should be able to track no less than two targets at the same time, should be able to continue tracking when the target is completely or partially occluded, and should be able to specify the target to be tracked using natural language.

[0035] The measurement sub-capabilities include distance measurement, velocity measurement, and object counting. The distance measurement system can measure the distance between two given points in an image, ideally the shortest surface distance between two given objects in an image, and the distance between an object and the observer. It should be able to perform measurements under different ambient lighting conditions and varying object surface reflectivity, with performance measured by mean absolute percentage error. The velocity measurement system can measure the moving speed of a given object in a video, ideally under conditions of occlusion, and ideally can simultaneously measure the moving speed of at least two objects. Performance is measured by mean absolute percentage error. The object counting system can count the number of objects of a given type in an image or video, ideally performing counts under conditions of occlusion and varying ambient lighting, with performance measured by mean absolute percentage error.

[0036] The recognition sub-capabilities include object recognition, optical character recognition, face recognition, content feature recognition, pose recognition, event or behavior recognition, and relationship recognition. The object recognition system should be able to identify specific individuals from images or videos, ideally under different viewing angles, lighting conditions, and occlusion conditions. Performance should be measured by recognition rate, false recognition rate, or missed recognition rate. The optical character recognition system should support printed and handwritten text recognition, and should be able to recognize text from at least five types of documents (such as ID cards, vehicle registration certificates, passports, bank cards, business licenses, etc.) and at least five types of invoices (such as VAT invoices, fixed-amount invoices, taxi receipts, train tickets, etc.). Ideally, it should be able to recognize characters in at least three languages. Performance should be measured by recognition rate, false recognition rate, or missed recognition rate. A facial recognition system should be able to detect faces in images or videos and return their locations. It should also be able to identify faces and return identifiers (such as names or IDs). Ideally, it should implement face comparison and liveness detection. Performance should be measured by recognition rate, false recognition rate, or missed recognition rate. Note that due to privacy and security requirements, facial recognition tasks should not directly return facial image data. A content feature recognition system should be able to identify and output content containing specific features (such as data, color, shape, etc.) from images, returning bounding boxes of relevant objects. Performance should be measured by accuracy. A pose recognition system should be able to detect key points (such as skeletons) of people or objects and output pose description information (such as the position of a person's head, hands, feet, etc.). Ideally, it should be able to output pose estimation descriptions for multiple objects or people in an image or video frame. Performance should be measured by average precision or key point similarity. An event or behavior recognition system should be able to identify predefined events or actions from images or videos. Ideally, it should be able to identify people or objects entering or leaving a specific area. Performance should be measured by accuracy and time intersection-over-union (TUI). Relationship recognition systems can identify spatial relationships (such as front-back relationships, top-bottom relationships, parallel relationships, etc.) between multiple objects, between people and objects, and between multiple people in images or videos. They should also be able to identify logical relationships (such as action interaction relationships, implicit relationships, etc.). The effectiveness is measured by the accuracy rate.

[0037] The visual expression sub-capabilities include image description, text-image understanding, multi-turn dialogue with visual content, mathematical calculation, and table comprehension. Image description systems should output words or natural language descriptions of given images or videos, with performance measured by subjective evaluation methods or the BLEU metric. Text-image understanding systems should understand mixed image and text inputs or images containing text, and should ideally be able to answer questions contained in the text-image input, with performance measured by accuracy. Multi-turn dialogue systems should engage in multi-turn interactions based on the content, type, or implied meaning of input images or videos, completing at least two rounds of interaction, and should ideally be able to complete dialogues using natural language, with performance measured by subjective evaluation methods and a sliding window dialogue relevance score. Mathematical calculation systems should perform calculations based on printed formulas contained in images or videos and output results, and should ideally be able to answer mathematical questions described in natural language, with performance measured by accuracy or subjective evaluation methods. Table comprehension systems should understand tables in given images or videos, and should ideally be able to extract necessary information from the tables to answer questions described in natural language or perform calculations, with performance measured by accuracy or subjective evaluation methods.

[0038] Visual generation capabilities encompass seven sub-capabilities: segmentation, completion, super-resolution, semantic expansion, replacement, combination, and removal.

[0039] The segmentation capabilities include instance segmentation and interactive segmentation. Instance segmentation systems can detect and output object contour information, and should ideally support custom descriptions of the objects to be segmented (e.g., using natural language). Performance is measured by average pixel accuracy and intersection-over-union (IoU). Interactive segmentation systems should support interactive methods such as clicking, drawing lines, or drawing boxes, and should ideally support at least two rounds of interaction. Performance is measured by minimum number of interactions, accuracy, and IoU.

[0040] The completion capabilities include image missing completion and video completion. The image missing completion system should be able to generate content to fill in missing areas according to rules and output the completed image. Ideally, it should be able to complete the completion while ensuring the missing area occupies at least 10% of the original image area. It should also be able to accept natural language input specifying the content or style of the completion area, and its effectiveness should be measured using a subjective evaluation method. The video completion system should be able to complete the generation and frame interpolation tasks (with a frame loss rate of at least 25%), and should be able to increase the given video frame rate to at least 24 frames per second. Its effectiveness should also be measured using a subjective evaluation method.

[0041] Super-resolution systems should achieve aspect ratio magnification of images or videos, preferably scaling up selected areas of the image or video proportionally, with a magnification ratio of no less than 10%. The effectiveness should be measured using subjective evaluation methods. Semantic extension systems should be able to extend input images along image edges or extend input videos forward and backward. They should be able to accept natural language or symbolic commands to specify the extension content, and their effectiveness should be measured using subjective evaluation methods.

[0042] The replacement capabilities include object replacement and style transfer. Object replacement systems should ideally be able to replace a specified region or object in an input image with a given object image, maintain logical consistency after replacement, and measure effectiveness through subjective evaluation methods. Style transfer systems should support at least one target style, ideally support video style transfer, and ideally support style descriptions using natural language, given images, or given videos, with effectiveness measured through subjective evaluation methods.

[0043] The combined capabilities include image stitching and fusion. An image stitching system should be able to determine the logical orientation of the input image and complete the stitching, ideally stitching images with overlapping content, and its effectiveness should be measured using image similarity or subjective evaluation methods. A fusion system should be able to integrate an image of a given object or background into the input image, ideally fusing images according to natural language descriptions, ideally maintaining the logical consistency of the fused image, and its effectiveness should be measured using subjective evaluation methods.

[0044] The removal system should be able to remove at least one object or person of a given category from the input image and complete the image, preferably be able to remove multiple objects or people of a given category, preferably be able to perform the removal operation according to natural language requirements, preferably be able to remove objects or people of a given category from the video and complete the video, and the effect should be measured by a subjective evaluation method.

[0045] like Figure 2 As shown, the evaluation method is executed as follows: After building the system in step S1, the corresponding test dataset is prepared according to the capabilities declared by the system under test in step S2; in step S3, the test platform drives the system under test to complete the test. The system under test receives requests through API remote calls, SDK local calls, or graphical user interfaces. The test platform and the system under test are deployed separately. Test samples are sent in a continuous single-arrival mode. The system under test does not participate in judging the correctness of the test samples, nor does it participate in the calculation of indicator values; in step S4, the objective evaluation tasks are calculated according to the corresponding formulas; in step S5, tasks requiring subjective judgment are calculated according to... Figure 3 The process shown completes the scoring; the final performance evaluation result is obtained by combining objective indicators and subjective scores in step S6.

[0046] like Figure 3 As shown, the specific process of subjective evaluation is as follows: An evaluation group is formed, and members are randomly divided into implementation groups (…). (person) and alternate group ( (People); each member of the implementation group is randomly and proportionally assigned a subset of test samples; tuples (input + output + reference value) are displayed for a specified duration p, and evaluators give a 5-point score within the scoring period g; the next tuple is displayed after a time interval d; evaluators whose overall evaluation rate is less than a certain percentage r are replaced by members of the reserve group; after completing the scoring using a single-round or multi-round subset method, the results are then... Calculate the subjective average total score, where For the first The scores given by each evaluator This represents the number of subjective ratings.

[0047] It should be noted that the specific methods by which each module performs operations in the system described in the above embodiments have been described in detail in the embodiments related to the method, and will not be elaborated here.

[0048] Finally, it should be noted that although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A multi-task visual perception and content processing evaluation method for complex scenarios, characterized in that, The method includes the following steps: Step S1: Build a multi-task visual perception and content processing system. The system is based on a four-layer reference architecture that includes an infrastructure layer, a data layer, an algorithm model layer, and an application layer. The system supports the processing of two major categories of tasks: visual understanding and visual generation through a large computer vision model in the algorithm model layer. Step S2: Based on the visual processing capabilities declared by the system under test, determine the corresponding evaluation tasks from the preset visual understanding task set and visual generation task set, and equip each evaluation task with a corresponding test dataset. Step S3: Send a test request to the system under test through the test platform and obtain the processing results of the system under test on the test sample; Step S4: For each evaluation task, select the corresponding objective evaluation indicators according to the task type, quantify the processing results, and obtain the values ​​of each objective performance indicator. Step S5: For evaluation tasks that require subjective judgment, an evaluation group is formed according to the prescribed subjective evaluation process to subjectively score the output results of the tested system and obtain a subjective evaluation score. Step S6: Combine the objective performance index values ​​with the subjective evaluation scores to obtain the comprehensive performance evaluation results of the tested system on each evaluation task.

2. The evaluation method according to claim 1, characterized in that, The visual understanding task set includes image classification, video classification, object detection, anomaly detection, rotation detection, change detection, distance measurement, velocity measurement, object counting, object recognition, optical character recognition, face recognition, content feature recognition, pose recognition, event or behavior recognition, relationship recognition, image description, image and text understanding, visual content multi-turn dialogue, mathematical calculation, and table understanding; the visual generation task set includes instance segmentation, interactive segmentation, image missing completion, video completion, super-resolution, semantic extension, object replacement, style transfer, image stitching, fusion, and removal.

3. The evaluation method according to claim 1, characterized in that, In step S2, the number of test dataset samples for each evaluation task shall not be less than 2000, and no less than 200 samples shall be randomly sampled from the corresponding dataset during a single test; when multiple systems to be tested are tested in the same batch, the same test set, sampling method and evaluation method shall be used.

4. The evaluation method according to claim 1, characterized in that, In step S4, the objective evaluation metrics include TOP1 accuracy, TOP5 accuracy, intersection-over-union ratio (IoU), recognition rate, false recognition rate, missed recognition rate, mean absolute percentage error, average precision, keypoint similarity, average pixel accuracy, image similarity, and sliding window dialogue relevance score. Specifically, image classification and video classification tasks use TOP1 accuracy or TOP5 accuracy; object detection and anomaly detection tasks use IoU or average precision; object recognition, optical character recognition, and face recognition tasks use recognition rate, false recognition rate, or missed recognition rate; distance measurement, velocity measurement, and object counting tasks use IoU; pose recognition tasks use average precision or keypoint similarity; instance segmentation tasks use average pixel accuracy and IoU; image stitching tasks use image similarity; and visual content multi-turn dialogue tasks use subjective evaluation methods and sliding window dialogue relevance score.

5. The evaluation method according to claim 4, characterized in that, TOP1 accuracy is obtained by calculating the ratio of the number of correct predictions to the total number of predictions; the intersection-union ratio is obtained by dividing the intersection area of ​​the labeled bounding box and the predicted bounding box by the union area of ​​the two; the mean absolute percentage error is obtained by averaging the ratio of the absolute value of the difference between the true value and the predicted value of each sample to the true value. Image similarity is achieved using the structural similarity method, which calculates the brightness, contrast, and structure of the stitched image and the ground truth image. The sliding window dialogue relevance score is obtained by averaging the relevance scores of each round of responses within a specified sliding window.

6. The evaluation method according to claim 1, characterized in that, In step S5, the evaluation group consists of n people, randomly divided into an implementation group and a reserve group. The number of people in the implementation group is equal to the number of test sample subsets. The evaluators should have the basic abilities required for evaluation and should preferably have basic knowledge of visual large model systems. For evaluators whose overall evaluation rate is less than the prescribed proportion, their evaluation results will not be included in the total score, and people will be selected from the reserve group to supplement the scoring.

7. The evaluation method according to claim 6, characterized in that, The subjective scoring process is as follows: a subset of test samples is randomly and proportionally assigned to each evaluator; for each test sample, the input, output, and reference values ​​are presented to the evaluators as a tuple, and the evaluators give a score within the specified scoring period; a single-round or multi-round subset scoring method is adopted, and when multiple rounds are the same subset, the average score of the multiple rounds is taken as the final score of the sample; the subjective average total score is calculated by averaging all valid scores.

8. A multi-task visual perception and content processing system for complex scenarios, characterized in that, The system comprises an infrastructure layer, a data layer, an algorithm model layer, and an application layer. The infrastructure layer includes computing devices, storage devices, network transmission devices, and visual signal acquisition and output devices, providing virtualization and task scheduling support. The data layer includes visual data such as images, videos, 3D models, image-text pairs, and video-text pairs, supporting visual data retrieval, synthesis, annotation, and quality assessment. The algorithm model layer is based on a large computer vision model, including pre-processing and post-processing components, supporting model compression and conversion, fine-tuning, and pre-training. The application layer supports workflow orchestration, preset template configuration, and visual model effect evaluation; the system supports parallel processing of two types of tasks: visual understanding and visual generation, and comprehensively evaluates the system performance using the evaluation method described in any one of claims 1 to 7.

9. The system according to claim 8, characterized in that, The visual understanding task includes five types of processing: classification, detection, measurement, recognition, and visual representation. Classification processing supports classifying images or videos under different viewing angles, lighting conditions, occlusion conditions, and resolution conditions. Detection processing supports object detection, anomaly detection, rotation detection, change detection, and object tracking, with rotation detection outputting a rotated bounding box. Measurement processing supports distance measurement, velocity measurement, and object counting. Recognition processing supports object recognition, optical character recognition, face recognition, content feature recognition, pose recognition, event or behavior recognition, and relationship recognition. Visual representation processing supports image description, text-image understanding, multi-turn dialogue of visual content, mathematical calculation, and table understanding.

10. The system according to claim 8, characterized in that, The visual generation task includes seven types of processing: segmentation, completion, super-resolution, semantic expansion, replacement, combination, and removal. Segmentation processing supports instance segmentation and interactive segmentation, with interactive segmentation supporting interactive methods such as clicking, drawing lines, or drawing frames. Completion processing supports image missing data completion and video completion, supporting completion of regular and irregular regions. Super-resolution processing supports aspect ratio magnification of images or videos, with a magnification ratio of no less than 10%. Replacement processing supports object replacement and style transfer. Combination processing supports image stitching and image or video fusion, maintaining the logical consistency of the fused content. Removal processing supports removing specified categories of objects or people from images or videos and completing the image.

Images