Classified scene video snapshot quality test system and method based on visual large model

Abstract

The invention provides a classification scene video snapshot quality test system and method based on a visual large model, and relates to the technical field of computer vision and quality assessment, and the method comprises the steps: carrying out the standardization processing of an input video snapshot, and extracting a scene class label contained in the video snapshot; performing feature extraction on the standardized video snapshots by adopting a pre-trained visual large model to obtain feature data of multiple levels; loading a feature weight matrix corresponding to a pre-constructed classification scene according to the classification scene determined by the scene category label, and calibrating the feature weight matrix to obtain a scene calibration weight matrix adaptive to the scene category label; and inputting the feature data of each level and the corresponding scene calibration weight matrix into a full-connection network to generate a quality test score. The method has the beneficial effects of improving scene universality, enhancing semantic understanding, improving dynamic adaptability and optimizing subjective consistency.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision and quality evaluation, and particularly relates to a classification scene video snapshot quality test system and method based on a visual large model. BACKGROUND

[0002] In today's digital era, the massive growth of video data makes video snapshot quality testing increasingly important. Whether in the fields of security monitoring, medical imaging, or autonomous driving, high-quality video snapshots are the foundation for ensuring the accuracy of subsequent analysis and decision-making. However, the current methods used for video snapshot quality testing have many problems, which seriously affect the reliability and practicality of the test results, mainly in the following aspects:

[0003] I. Poor scene adaptability

[0004] Different classification scenes have their own unique visual features and requirements. For example, in the security monitoring scene, the focus is on the integrity and recognizability of the target; in the medical imaging scene, the details and clarity of the organization are highly required; and in the autonomous driving scene, accurate capture of road, traffic signs, and pedestrians is needed. Existing video snapshot quality testing mainly relies on traditional image processing algorithms, which evaluate quality based on artificially designed features such as edges, textures, contrast, etc. Therefore, special features need to be designed for each specific scene, which not only consumes a lot of manpower and time cost, but also requires re-designing features and adjusting algorithms when facing new scenes or changes in scene characteristics, greatly limiting the method's promotion and application range.

[0005] II. Insufficient semantic understanding

[0006] Traditional methods only stay at the pixel level or low-level feature level when evaluating quality, lacking semantic analysis of video snapshot content. Semantic information is crucial for accurately evaluating video snapshot quality. For example, in the security scene, the integrity of the target object in a video snapshot is one of the key factors in evaluating its quality. If only low-level features such as edges and textures are used for evaluation, it may not accurately determine whether the target object is complete, thus affecting the overall evaluation of snapshot quality.

[0007] III. Weak dynamic updating capability

[0008] Existing video snapshot quality testing techniques are mostly static evaluation models that cannot automatically adjust their parameters to adapt to new scenes or changes in data distribution, requiring frequent manual parameter adjustment. This not only is inefficient, but also requires high professional requirements for operators. In practical applications, manual parameter adjustment often cannot keep up with the speed of data changes, resulting in a significant decline in the evaluation accuracy of the model in new scenes, which cannot meet the requirements of real-time and accuracy.

[0009] IV. Low subjective consistency

[0010] Human visual perception is a complex process influenced by various factors, including lighting, color, object form, and individual experience and expectations. Traditional methods are based on manually designed features and fixed algorithms for evaluation, resulting in poor consistency with human visual perception, especially in complex scenes.

[0011] In the field of automated testing, some solutions attempt to use shallow neural networks for quality classification, but due to limited model capacity, they cannot effectively extract high-level semantic features, and have weak recognition ability for fine-grained quality differences (such as slight blur, color distortion), making it difficult to meet the demand for high-precision evaluation of video snapshot quality in practical applications.

[0012] Therefore, it is necessary to improve the existing video snapshot quality testing method. SUMMARY

[0013] To solve the problems in the prior art, the present application provides a classification scene video snapshot quality testing system based on a visual large model, comprising:

[0014] A data preprocessing module is used to standardize the input video snapshot and extract the scene category label contained therein;

[0015] A feature extraction module is connected to the data preprocessing module and is used to extract features from the standardized video snapshot using a pre-trained visual large model, obtaining multiple levels of feature data;

[0016] A scene adaptive calibration module is connected to the data preprocessing module and is used to load a pre-constructed feature weight matrix corresponding to the classification scene according to the classification scene determined by the scene category label, and calibrate the feature weight matrix to obtain a scene calibration weight matrix adapted to the scene category label;

[0017] A quality score generation module is connected to the feature extraction module and the scene adaptive standard module, respectively, and is used to input each level of feature data and its corresponding scene calibration weight matrix into a fully connected network to generate a quality testing score.

[0018] Preferably, the standardization process includes one or more of resolution unification, color space conversion, and noise filtering.

[0019] Preferably, the multiple levels of feature data include low-level pixel features, middle-level semantic features, and high-level scene features, which are one-to-one corresponding to respective weight values in the feature weight matrix.

[0020] Preferably, the scene adaptive calibration module calibrates the feature weight matrix by meta-learning, or a rule-based heuristic method, or a preset feature priority in different classification scenes.

[0021] Preferably, the quality score generation module includes a score generation unit for weighting the feature data of each level according to the scene calibration weight matrix and concatenating in the channel dimension, and then inputting into the two-layer fully connected network to generate the quality test score.

[0022] Preferably, the quality score generation module includes a score mapping unit for dividing a preset score range into multiple sub-ranges, each sub-range being associated with a corresponding quality level label, and mapping the generated quality test score to the quality level label.

[0023] Preferably, it further includes a dynamic weight calibration module connected to the data preprocessing module and the quality score generation module, respectively, the dynamic weight calibration module including:

[0024] A first acquisition unit for acquiring a plurality of historical video snapshots for each classification scene, each historical video snapshot corresponding to an artificial annotation quality score;

[0025] A second acquisition unit for inputting each historical video snapshot into the data preprocessing module and obtaining the quality test score generated by the quality score generation module as a model prediction score;

[0026] A weight calibration unit connected to the first acquisition unit and the second acquisition unit, respectively, for dynamically calibrating the feature weight matrix by minimizing the mean square error between the artificial annotation quality score and the model prediction score in each classification scene.

[0027] Preferably, the mean square error between the artificial annotation quality score and the model prediction score is minimized by a minimization error function, and the expression of the minimization error function is as follows:

[0028]

[0029] wherein W S * represents the calibrated feature weight matrix, W SW represents the feature weight matrix pre-constructed when the classification scene is S S =[w1,w2,…,wn], wn represents the weight of the nth feature dimension in the feature weight matrix, m represents the sample number of the historical video snapshot when the classification scene is S, X j represents the jth sample when the classification scene is S, y j represents the artificial annotation quality score of the jth sample, f(X j ; W S ) represents the model prediction score.

[0030] Preferably, it further comprises an online learning module connected with the data preprocessing module and the dynamic weight calibration module respectively, for continuously counting the scene category label of the input video snapshot, and generating an update signal when the scene category label is a new scene, and generating an update signal when the count value reaches a preset value;

[0031] The dynamic weight calibration module adds each video snapshot accumulated as incremental data into the historical video snapshot according to the update signal, thereby triggering dynamic calibration of the feature weight matrix.

[0032] The application also provides a classification scene video snapshot quality testing method based on a visual large model, applied to the classification scene video snapshot quality testing system, and the classification scene video snapshot quality testing method comprises the following steps:

[0033] Step S1, standardizing the input video snapshot and extracting the scene category label contained therein;

[0034] Step S2, using a pre-trained visual large model to extract features from the standardized video snapshot, to obtain multiple levels of feature data;

[0035] Step S3, loading the feature weight matrix corresponding to the classification scene pre-constructed according to the classification scene determined by the scene category label, and calibrating the feature weight matrix to obtain a scene calibration weight matrix adapted to the scene category label;

[0036] Step S4, inputting each level of feature data and its corresponding scene calibration weight matrix into a full connection network to generate a quality testing score.

[0037] The above technical solution has the following advantages or beneficial effects:

[0038] 1. Improved scene universality: through the cross-scene generalization ability of the visual large model, support for multiple classification scenes such as security, medical treatment, and autonomous driving, without the need for separate modeling for a single scene, and the development cost is reduced by more than 60%.

[0039] 2. Semantic understanding enhancement: Through multi-level feature extraction, the quality evaluation accuracy of key targets (such as faces in security and lesions in medical treatment) in video snapshots is improved to 92% (increased by 35% compared with traditional methods).

[0040] 3. Dynamic adaptability: The adaptation time of new scenarios is shortened from 72 hours in traditional methods to 2 hours, supporting real-time online updates and significantly improving test efficiency.

[0041] 4. Subjective consistency optimization: The Spearman correlation coefficient with human subjective scoring reaches 0.91, close to the level of professional manual evaluation, effectively reducing the misjudgment rate. BRIEF DESCRIPTION OF DRAWINGS

[0042] Figure 1 For the preferred embodiment of the present application, a structure diagram of a classification scene video snapshot quality test system based on a visual large model is provided.

[0043] Figure 2 For the preferred embodiment of the present application, a flowchart of a classification scene video snapshot quality test method based on a visual large model is provided. DETAILED DESCRIPTION

[0044] The present application will be described in detail below in conjunction with the drawings and specific embodiments. The present application is not limited to this embodiment, and other embodiments can also fall within the scope of the present application as long as they comply with the spirit of the present application.

[0045] In the preferred embodiment of the present application, based on the above-mentioned problems existing in the prior art, a classification scene video snapshot quality test system based on a visual large model is provided, as shown in Figure 1 , comprising:

[0046] The data preprocessing module 1 is used for standardizing the input video snapshot and extracting the scene category label contained therein.

[0047] The feature extraction module 2 is connected to the data preprocessing module 1 and is used for extracting features from the standardized video snapshot using a pre-trained visual large model to obtain multiple levels of feature data.

[0048] The scene adaptive calibration module 3 is connected to the data preprocessing module 1 and is used for loading the pre-constructed feature weight matrix corresponding to the classification scene according to the classification scene determined by the scene category label, and calibrating the feature weight matrix to obtain the scene calibration weight matrix of the adaptive scene category label.

[0049] The quality score generation module 4 is connected to the feature extraction module 2 and the scene adaptive standard module 3 respectively, and is used for inputting the feature data of each level and the corresponding scene calibration weight matrix into a full connection network to generate a quality test score.

[0050] Specifically, the present application realizes the automatic and accurate quality evaluation of the classification scene video snapshot by constructing a visual large model driven quality test framework. The overall framework design includes data preprocessing module 1, feature extraction module 2, scene adaptive calibration module 3 and quality score generation module 4.

[0051] Among them, the data preprocessing module 1 performs standardization processing on the input video snapshot, including one or more of resolution unification, color space conversion, noise filtering, while extracting basic metadata, here the basic metadata includes the shooting time, device model and scene category label corresponding to the video snapshot. Among them, the video snapshot based on different device models will create corresponding scene category labels, which can be batch preprocessed according to the shooting time to obtain the corresponding scene category labels.

[0052] The feature extraction module 2 adopts a pre-trained visual large model (such as ViT-G, Swin-Large) to perform multi-level feature extraction on the standardized video snapshot, and the multi-level feature data includes:

[0053] Low-level pixel features: pixel-level features are extracted by a convolutional neural network (CNN), including but not limited to edges, textures and color distributions.

[0054] Mid-level semantic features: local region semantic features are captured as mid-level semantic features using a Transformer architecture, including but not limited to object contours and action poses.

[0055] High-level scene features: combined with the scene category label, scene-level semantic features are extracted as high-level scene features through cross-modal fusion (such as text-visual alignment), such as "target integrity" in the security scene and "lesion clarity" in the medical scene.

[0056] As a preferred embodiment, the visual large model can be processed to obtain a lightweight model (such as MobileViT), so that the present application can be deployed on a mobile terminal or in a power limited scene. When performing lightweight processing, it is preferred to retain the core quality evaluation capability through knowledge distillation.

[0057] The scene adaptive calibration module 3 loads the feature weight matrix corresponding to the classification scene to which the current input video snapshot belongs based on the scene category label of the current input video snapshot. The feature weight matrix is obtained by pre-construction, and the weight values in the feature weight matrix correspond to the low-level pixel features, the mid-level semantic features and the high-level scene features one by one.

[0058] Further, to improve the accuracy of quality testing, the scene adaptive calibration module 3 also combines a dynamic calibration mechanism to calibrate the feature weight matrix to obtain a scene calibration weight matrix adapted to the scene category label, so as to dynamically adjust the feature fusion strategy, so that the quality score result is closer to human subjective perception, and the artificial intervention cost is reduced.

[0059] Among them, the feature weight matrix is the quality sensitive feature weight under each classification scene calculated from the historical annotation data, such as "target resolution" weight higher than "color saturation" in the security scene. Based on this, it also includes a dynamic weight calibration module 5, which is respectively connected to the data preprocessing module 1 and the quality score generation module 4, and the dynamic weight calibration module 5 includes:

[0060] The first acquisition unit 51 is used to acquire a plurality of historical video snapshots for each classification scene, and each historical video snapshot corresponds to an artificial annotation quality score, that is, to form historical annotation data;

[0061] The second acquisition unit 52 is used to input each historical video snapshot into the data preprocessing module, and acquire the quality test score generated by the quality score generation module as the model prediction score;

[0062] The weight calibration unit 53 is respectively connected to the first acquisition unit 51 and the second acquisition unit 52, and is used to dynamically calibrate the feature weight matrix by minimizing the mean square error between the artificial annotation quality score and the model prediction score under each classification scene.

[0063] In the preferred embodiment of the application, the mean square error between the artificial annotation quality score and the model prediction score is minimized by minimizing the error function, and the expression of the error function is as follows:

[0064]

[0065] Among them, W S * represents the calibrated feature weight matrix, W S represents the feature weight matrix constructed in advance when the classification scene is S, W S =[w1,w2,…,wn], wn represents the weight of the n th feature dimension in the feature weight matrix, m represents the sample number of the historical video snapshot when the classification scene is S, X j represents the j th sample when the classification scene is S, y j represents the artificial annotation quality score of the j th sample, f(X j ; W S ) represents the model prediction score.

[0066] Further, when the scene data is input, the feature weight matrix is updated by incremental training to avoid full training, thereby improving the adaptation efficiency. Specifically, the online learning module 6 is connected to the data preprocessing module 1 and the dynamic weight calibration module 5, respectively, for continuously counting the scene category labels of the input video snapshots, and counting once when the scene category label is a new scene, and generating an update signal when the count value reaches a preset value.

[0067] The dynamic weight calibration module 5 adds the cumulative input video snapshots as incremental data to the historical video snapshots according to the update signal, thereby triggering the dynamic calibration of the feature weight matrix.

[0068] Specifically, in the embodiment, the distinction between "new scene" and "classified scene" is the core function of the online learning module. The following explains in detail from two aspects of technical implementation and business logic:

[0069] Conceptual distinction and determination logic

[0070] 1. Classified scene (predefined scene)

[0071] Definition: Scene categories preset when the system is initialized, based on the pre-training classification system of the visual large model

[0072] Examples: security scene (which can be further subdivided into monitoring, access control, perimeter defense, etc.)

[0073] Features:

[0074] Has a clear hierarchical classification system

[0075] Each scene has a preset feature priority matrix (as described in claim 7)

[0076] Quality evaluation standards are fixed in the system

[0077] 2. New scene (unidentified scene)

[0078] Definition: Scene type discovered during operation that is not covered by the predefined classification system

[0079] Determination conditions:

[0080] The visual large model classification confidence is below the threshold value (e.g., <0.6)

[0081] The feature distribution is less than the threshold value similar to all predefined scenes

[0082] The number of consecutive occurrences reaches a preset value (e.g., 20 times)

[0083] Examples:

[0084] Predefined scene: security scene (including monitoring, traditional access control)

[0085] New scenes:

[0086] New model smart access control (such as iris recognition access control)

[0087] Special application scenarios

[0088] Cross-domain scenarios (such as AGV access control in industrial security).

[0089] In a preferred embodiment of the present invention, the scene adaptive calibration module 3 calibrates the feature weight matrix by using meta-learning, or a rule-based heuristic method, or by presetting feature priorities under different classification scenarios.

[0090] Specifically, in this embodiment, the rule-based heuristic method requires calibrating the weight matrices of multiple features such as "clarity", "color consistency", "brightness stability", and "object integrity". The heuristic calibration logic based on the video classification scenario is specifically implemented as follows:

[0091] First, a feature weight matrix is ​​preset, which contains the initial weight values ​​of four video quality assessment features, namely [clarity, color consistency, brightness stability, and object integrity].

[0092] Then, a series of preset rules are applied to adjust the feature weight matrix according to the video scene type and attributes. The preset rules include scene type rules and video attribute rules. The scene type rules include:

[0093] For landscape videos (scene_type="landscape"), the weights for color consistency (40%) and brightness stability (30%) can be increased, as landscape videos have higher requirements for color and lighting performance.

[0094] For interview videos (scene_type="interview"), the weights of clarity (50%) and object integrity (20%) can be increased because interview videos need to highlight facial details.

[0095] Video attribute rules include:

[0096] For low-resolution videos (resolution < 720), force the clarity weight to 60% and renormalize other weights, because clarity is the most critical factor at low resolutions;

[0097] High dynamic scenes (motion_intensity>0.8): Substantially increase the brightness stability weight (60%), because brightness changes in dynamic scenes are more easily perceived.

[0098] Normalization is performed after each adjustment to ensure that all weights sum to 1, maintaining the normalization of the weight matrix.

[0099] Through the above rule-based heuristic method, it has the following characteristics:

[0100] Domain knowledge driven: Rules are designed based on domain knowledge of video quality assessment.

[0101] Dynamic adaptation: Automatically adjust evaluation criteria according to different video types and attributes.

[0102] Scalability: New scene types or video attribute rules can be easily added.

[0103] It realizes the adaptive adjustment of different types of video quality evaluation criteria, making the evaluation results more in line with the actual perception needs.

[0104] Examples are as follows:

[0105] The initial weight distribution is: [clarity 30%, color consistency 25%, brightness stability 25%, object integrity 20%].

[0106] Video metadata is video resolution 1080P, motion intensity evaluation 0.9 (high motion intensity), frame rate 30 frames per second.

[0107] If the current scene type is a landscape scene, the following calibration is triggered:

[0108] Landscape scene rules: color weight x 1.4, brightness weight x 1.3

[0109] High dynamic scene rules: brightness weight additional x 1.6

[0110] Resolution rules: not triggered (because 1080>720)

[0111] The final output is the feature weight value after normalization after these rule adjustments.

[0112] Based on the preset feature priority method, for different video classification scenes, the preset feature priority matrix is combined with the basic weight output by the visual large model to perform the following calibration processing logic:

[0113] 1. Scene priority matrix

[0114] Predefine the feature weight priority of four typical scenes:

[0115] Landscape: emphasize color consistency (30%) and brightness stability (25%)

[0116] Sports: emphasize brightness stability (30%) and clarity (25%)

[0117] Interview: emphasize clarity (35%) and object integrity (30%)

[0118] Animation: emphasize color consistency (35%) and semantic accuracy (20%)

[0119] 2. Confidence adjustment mechanism:

[0120] Use confidence_score (0-1 range) to control calibration strength;

[0121] The higher the confidence (confidence_score close to 1), the greater the calibration strength, and the closer the final weight to the scene priority.

[0122] Confidence conversion formula: calibration_strength = min(1.0, confidence_score * 1.5)

[0123] 3. Weight fusion algorithm

[0124] Calibrated weight = base weight[i] × (1 - calibration strength) + scene priority weight[i] × calibration strength

[0125] The essence of this algorithm formula is a linear interpolation (Lerp) operation. When the calibration strength is 0 (low confidence), the original base weight is completely retained; when the calibration strength is 1 (high confidence), the scene recommended weight is completely adopted; and the intermediate value is a proportional mixture of the two weights.

[0126] Example as follows:

[0127] If the base weight output by the visual large model is [0.25, 0.25, 0.25, 0.2, 0.2, 0.1], the current video is classified as sports, and the classification confidence is 85%.

[0128] First, get the sports priority: [0.25, 0.15, 0.3, 0.2, 0.1]

[0129] Calculate the calibration strength: 0.85 * 1.5 = 1.275 → take 1.0 (upper limit)

[0130] Because of high confidence, the final weight will completely adopt the sports scene priority, and the output result should be the same as the sports preset priority matrix, especially strengthening the weight of brightness stability (30%), which meets the requirements of sports video for dynamic picture stability.

[0131] In the preferred embodiment of the present application, the quality score generation module 4 comprises a score generation unit 41 for weighting the feature data of each level according to the scene calibration weight matrix and splicing in the channel dimension, and then inputting into a two-layer fully connected network to generate the quality test score.

[0132] Specifically, in the present embodiment, the score generation unit 41 preferably splices the bottom layer pixel features, the middle layer semantic features and the high layer scene features in the channel dimension through a feature pyramid fusion structure, retains different granularity information, and realizes multi-granularity feature fusion.

[0133] In the preferred embodiment of the present application, the quality score generation module 4 comprises a score mapping unit 42 for dividing a preset score range into a plurality of sub-ranges, associating each sub-range with a corresponding quality level label, and mapping the generated quality test score into the quality level label.

[0134] Specifically, in the present embodiment, the preset score range can be 0-100 points, the corresponding sub-ranges can include four, and the corresponding quality level labels can correspond to excellent, good, medium and poor.

[0135] The present application also provides a classification scene video snapshot quality test method based on a visual large model, applied to the classification scene video snapshot quality test system described above, as shown in Figure 2 The classification scene video snapshot quality test method comprises the following steps:

[0136] Step S1, standardizing the input video snapshot and extracting the scene category label contained therein;

[0137] Step S2, extracting features from the standardized video snapshot by using a pre-trained visual large model to obtain a plurality of levels of feature data;

[0138] Step S3, loading the feature weight matrix corresponding to the classification scene according to the classification scene determined by the scene category label, and calibrating the feature weight matrix to obtain a scene calibration weight matrix adapted to the scene category label;

[0139] Step S4, inputting the feature data of each level and the corresponding scene calibration weight matrix into a fully connected network to generate a quality test score.

[0140] Embodiment I

[0141] The application of the present application to the security scene video snapshot quality test process is as follows:

[0142] 1. Data preprocessing

[0143] Input: A snapshot of a security monitoring video with resolution 1920x1080, containing metadata (scene category label = "access control monitoring", device = "Hikvision DS-2CD3T46WD").

[0144] Processing: Uniformly scale the resolution to 512x512, convert to YUV color space, remove salt and pepper noise through median filtering.

[0145] 2. Feature Extraction

[0146] Low-level pixel features: Extract edge and texture features from RGB channels using ResNet50, output a 2048-dimensional vector.

[0147] Mid-level semantic features: Process the blocked image patches using ViT-Base model to capture local semantics such as faces and access control devices, output a 1024-dimensional vector.

[0148] High-level scene features: Combine the scene category label "access control monitoring" and extract scene semantic features such as "face clarity" and "access control sign integrity" using a pre-trained image-text alignment model (e.g., CLIP), output a 512-dimensional vector.

[0149] 3. Scene Adaptive Calibration

[0150] Load the pre-trained weight matrix for security scenes W_security = [0.3, 0.4, 0.3] (corresponding to the weights of low-level pixel features, mid-level semantic features, and high-level scene features, respectively).

[0151] If the current data is a new access control scene, perform fast meta-learning with 5 labeled samples to adjust the weights to W_new_security = [0.25, 0.5, 0.25], enhancing the influence of mid-level semantic features.

[0152] 4. Quality Score Generation

[0153] Fuse feature vectors: F = [low-level pixel features; mid-level semantic features; high-level scene features], input into two fully connected networks (hidden layer dimensions 1024, 512).

[0154] Output score: Map to 0-100 points through the Sigmoid function, if the score ≥ 85 points, label "excellent", 60-85 points "good", 40-60 points "medium", <40 points "poor".

[0155] 5. Online Update

[0156] When 100 new scene samples are accumulated, trigger incremental training to update the scene calibration module, maintaining the model's evaluation accuracy for new access control device snapshots.

[0157] Example Two

[0158] Apply the application to the medical image scene, on the basis of embodiment one, the following differentiated processing is carried out:

[0159] 1. Data difference

[0160] The input is a CT image snapshot in DICOM format. Window width and window level adjustment is required during preprocessing to preserve the professional features of medical images.

[0161] 2. Feature optimization

[0162] In high-level feature extraction, a pre-trained visual large model in the medical field (such as MedViT) is introduced to enhance semantic understanding of the lesion area.

[0163] 3. Calibration strategy

[0164] In the weight matrix of the medical scene, the feature weight proportion of "lesion edge definition" and "noise level" is increased to 60%, ensuring the priority of quality evaluation of key diagnostic information.

[0165] Embodiment three

[0166] Lightweight deployment scheme of the application

[0167] 1. Model compression

[0168] Pruning and quantization are performed on the visual large model to reduce the parameter quantity from 10B to less than 1B, adapting to mobile devices.

[0169] 2. Calibration simplification

[0170] A lightweight calibration module based on rules is used to preset the feature priority in different scenarios, achieving fast quality testing in limited computing power environments.

[0171] The above description is only the preferred embodiment of the application, and does not limit the implementation and protection scope of the application. For those skilled in the art, it should be realized that any equivalent replacement and obvious changes made by applying the content of the specification and drawings should be included in the protection scope of the application.

Claims

1. A classification scene video snapshot quality testing system based on a large visual model, characterized by: include: The data preprocessing module is used to normalize the input video snapshots and extract the scene category labels contained therein; A feature extraction module, connected to the data preprocessing module, is used to extract features from the normalized video snapshots using a pre-trained visual model to obtain feature data at multiple levels; A scene adaptive calibration module, connected to the data preprocessing module, is used to load a pre-built feature weight matrix corresponding to the classification scene according to the classification scene determined by the scene category label, and calibrate the feature weight matrix to obtain a scene calibration weight matrix adapted to the scene category label; The quality score generation module is connected to the feature extraction module and the scene adaptation standard module respectively, and is used to input the feature data of each level and the corresponding scene calibration weight matrix into a fully connected network to generate a quality test score.

2. The classification scene video snapshot quality testing system according to claim 1, characterized in that: The standardization process includes one or more of resolution unification, color space conversion, and noise filtering.

3. The classification scene video snapshot quality testing system according to claim 1, characterized in that: The feature data at multiple levels include bottom-level pixel features, middle-level semantic features and high-level scene features, and the bottom-level pixel features, the middle-level semantic features and the high-level scene features correspond one-to-one to each weight value in the feature weight matrix.

4. The classification scene video snapshot quality testing system according to claim 1, characterized in that: The scene adaptive calibration module calibrates the feature weight matrix by using meta-learning, or a rule-based heuristic method, or by presetting feature priorities under different classification scenarios.

5. The classification scene video snapshot quality testing system according to claim 1, characterized in that: The quality score generation module includes a score generation unit, which is used to weight the feature data of each layer according to the scene calibration weight matrix and splice it according to the channel dimension, and then input it into the two layers of the fully connected network to generate the quality test score.

6. The classification scene video snapshot quality testing system according to claim 1, characterized in that: The quality score generation module includes a score mapping unit, which is used to divide a preset score range into multiple sub-ranges, associate each sub-range with a corresponding quality grade label, and map the generated quality test score to the quality grade label.

7. The classification scene video snapshot quality testing system according to claim 1, characterized in that: It also includes a dynamic weight calibration module, which is connected to the data preprocessing module and the quality score generation module respectively, and the dynamic weight calibration module includes: A first acquisition unit is configured to acquire a plurality of historical video snapshots for each of the classification scenes, each of the historical video snapshots being annotated with a manually annotated quality score; A second acquisition unit is configured to input each of the historical video snapshots into the data preprocessing module, and acquire the quality test score generated by the quality score generation module as a model prediction score; A weight calibration unit is connected to the first acquisition unit and the second acquisition unit respectively, and is used to dynamically calibrate the feature weight matrix by minimizing the mean square error between the manual annotation quality score and the model prediction score in each classification scenario.

8. The classification scene video snapshot quality testing system according to claim 7, characterized in that: The mean square error between the manual annotation quality score and the model prediction score is minimized using a minimization error function. The expression of the minimization error function is as follows: Among them, W S * Represents the calibrated feature weight matrix, W S represents the feature weight matrix pre-constructed when the classification scenario is S, W S =[w1,w2,…,wn], where wn represents the weight of the nth feature dimension in the feature weight matrix, m represents the number of samples of the historical video snapshot when the classification scenario is S, and X j Indicates the jth sample when the classification scenario is S, y j represents the manual annotation quality score of the jth sample, f(X j ;W S ) represents the model prediction score.

9. The classification scene video snapshot quality testing system according to claim 7, characterized in that: The system further includes an online learning module, connected to the data preprocessing module and the dynamic weight calibration module, respectively, for continuously counting the scene category labels of the input video snapshots, counting once when the scene category label is a new scene, and generating an update signal when the count value reaches a preset value; The dynamic weight calibration module adds the accumulated input video snapshots as incremental data to the historical video snapshots according to the update signal, thereby triggering the dynamic calibration of the feature weight matrix.

10. A method for testing the quality of classified scene video snapshots based on a large visual model, characterized in that: The classification scene video snapshot quality testing system according to any one of claims 1 to 9 is applied, wherein the classification scene video snapshot quality testing method comprises: Step S1, normalize the input video snapshot and extract the scene category labels contained therein; Step S2, using a pre-trained visual model to extract features from the normalized video snapshot to obtain feature data at multiple levels; Step S3, loading a pre-built feature weight matrix corresponding to the classification scene according to the classification scene determined by the scene category label, and calibrating the feature weight matrix to obtain a scene calibration weight matrix adapted to the scene category label; Step S4: input the feature data of each level and the corresponding scene calibration weight matrix into a fully connected network to generate a quality test score.

Images