Training method of video quality evaluation model, video quality evaluation method and model

Abstract

This application discloses a training method, a video quality assessment method, and a model for a video quality assessment model. The method includes obtaining a training dataset comprising at least one training sample group; each training sample group includes a sequence of sample video frames and a set of descriptive text corresponding to the sample video frame sequence; inputting a training sample group into the video quality assessment model to be trained, which generates motion distortion loss, semantic feature loss, and / or semantic consistency loss corresponding to the training sample group; adjusting the model parameters of the video quality assessment model to be trained based on the motion distortion loss, semantic feature loss, and / or semantic consistency loss corresponding to a preset number of training sample groups; and repeatedly executing the training process using the training sample groups in the training dataset until a preset termination condition is met, thereby obtaining the trained video quality assessment model. This method can train a video quality assessment model with strong generalization ability.

Description

Technical Field

[0001] This application relates to the field of computer technology, specifically to a training method and apparatus for a video quality assessment model, a video quality assessment method and apparatus, a video quality assessment model, a video-based semantic distortion assessment model, an electronic device, a computer-readable storage medium, and a computer program product. Background Technology

[0002] With the development of mobile internet, UGC (user-generated content) short video platforms have rapidly emerged and become an important part of people's daily lives. However, the explosive growth in the number of UGC short videos has made the effective evaluation of their quality an urgent problem to be solved.

[0003] Currently, video quality assessment methods are broadly categorized into full-reference and no-reference methods based on whether a lossless video reference is needed. Since no-reference video quality assessment methods do not require a lossless video reference, they have wider applicability. However, existing no-reference video quality assessment models are only applicable to single scenes, exhibit significant performance degradation across datasets, and have poor generalization ability. Furthermore, existing no-reference video quality assessment models show uneven sensitivity to different distortion types, making no-reference quality prediction tasks extremely challenging. Therefore, exploring objective no-reference video quality assessment methods is particularly important. Summary of the Invention

[0004] This application provides a training method and apparatus for a video quality assessment model, a video quality assessment model, a video-based semantic distortion assessment model, an electronic device, a computer-readable storage medium, and a computer program product. It is capable of training a video quality assessment model with strong generalization ability, applicable to assessing the quality of videos with various complex distortion types, for no-reference quality prediction tasks. The specific solution is as follows:

[0005] Firstly, this application provides a method for training a video quality assessment model, the method comprising:

[0006] Obtain the training dataset; the training dataset includes at least one training sample group, each training sample group including at least: a sample video frame sequence and a set of descriptive text corresponding to the sample video frame sequence; the set of descriptive text includes at least one pre-defined video quality descriptive text used to represent high-quality semantics; train the video quality assessment model to be trained based on the training dataset using the following training process: input a training sample group from the training dataset into the video quality assessment model to be trained, the motion distortion assessment module, the semantic distortion assessment module, and / or the semantic consistency assessment module generate the motion distortion loss, semantic feature loss, and / or semantic consistency loss corresponding to the training sample group, respectively; the motion distortion loss is used to describe the motion state of a pre-defined object in the sample video frame sequence. The training process involves several steps: First, a motion distortion loss is applied to the video quality assessment model. Second, a semantic feature loss is applied to describe the semantic discriminability of the video frame sequences. Third, a semantic consistency loss is applied to describe the semantic alignment between the video frame sequences corresponding to the training sample groups and the quality description texts in the corresponding description text sets. Finally, the training parameters are adjusted based on a preset number of training sample groups, including the motion distortion loss, semantic feature loss, and / or semantic consistency loss. The training process is repeated using a sufficient number of training sample groups from the training dataset until the preset end-of-training conditions are met, resulting in the trained video quality assessment model.

[0007] Secondly, this application also provides a video quality assessment method, which includes: acquiring a target video frame sequence to be assessed; inputting the target video frame sequence into a video quality assessment model so that the video quality assessment model outputs a quality score corresponding to the target video frame sequence; wherein the video quality assessment model is trained according to the training method provided in the first aspect.

[0008] Thirdly, this application also provides a video quality assessment method, which includes: performing content reconstruction processing on a target video frame sequence to be quality assessed to obtain a reconstructed video frame sequence corresponding to the target video frame sequence; determining the semantic feature loss corresponding to the target video frame sequence based on the target video frame sequence and the reconstructed video frame sequence; the semantic feature loss is used to describe the degree of influence of the semantic discriminability of the sample video frame sequence on the quality of the sample video frame sequence; and determining the target quality score corresponding to the target video frame sequence based on the semantic feature loss.

[0009] Fourthly, this application also provides a training device for a video quality assessment model, wherein the video quality assessment model to be trained includes at least: a motion distortion assessment module, a semantic distortion assessment module, and / or a semantic consistency assessment module; the device includes: an acquisition unit and a training unit;

[0010] The acquisition unit is used to acquire the training dataset; the training dataset includes at least one training sample group, and each training sample group includes at least: a sample video frame sequence and a set of descriptive text corresponding to the sample video frame sequence; the set of descriptive text includes at least one pre-defined video quality descriptive text used to represent high-quality semantics;

[0011] The training unit trains the video quality assessment model to be trained based on the training dataset using the following process: A training sample group from the training dataset is input into the video quality assessment model. The motion distortion assessment module, semantic distortion assessment module, and / or semantic consistency assessment module generate motion distortion loss, semantic feature loss, and / or semantic consistency loss corresponding to the training sample group, respectively. Motion distortion loss describes the impact of the motion state of a preset object in the sample video frame sequence on the quality of the sample video frame sequence. Semantic feature loss describes the impact of the semantic discriminability of the sample video frame sequence on the quality of the sample video frame sequence. Semantic consistency loss describes the impact of the semantic alignment between the sample video frame sequence corresponding to the training sample group and the quality description text in the set of description text corresponding to the sample video frame sequence on the quality of the sample video frame sequence. The model parameters of the video quality assessment model to be trained are adjusted according to the motion distortion loss, semantic feature loss, and / or semantic consistency loss corresponding to a preset number of training sample groups. The training process is repeated using a sufficient number of training sample groups from the training dataset until the preset end-of-training condition is met, resulting in the trained video quality assessment model.

[0012] Fifthly, this application also provides a video quality assessment apparatus, which includes: an acquisition unit and a generation unit;

[0013] The acquisition unit is used to acquire the target video frame sequence to be evaluated for quality.

[0014] The generation unit is used to input the target video frame sequence into the video quality assessment model so that the video quality assessment model outputs the quality score corresponding to the target video frame sequence; the video quality assessment model is trained according to the training method provided in the first aspect.

[0015] Sixthly, this application also provides a video quality assessment apparatus, the apparatus comprising: a reconstruction unit and a determination unit;

[0016] The reconstruction unit is used to perform content reconstruction processing on the target video frame sequence to be quality assessed, so as to obtain the reconstructed video frame sequence corresponding to the target video frame sequence.

[0017] The determination unit is used to determine the semantic feature loss corresponding to the target video frame sequence based on the target video frame sequence and the reconstructed video frame sequence; the semantic feature loss is used to describe the degree to which the semantic discriminability of the sample video frame sequence affects the quality of the sample video frame sequence.

[0018] The determination unit is also used to determine the target quality score corresponding to the target video frame sequence based on semantic feature loss.

[0019] In a seventh aspect, this application also provides an electronic device, comprising: a processor, a memory, and computer program instructions stored in the memory and executable on the processor; wherein the processor executes the computer program instructions to implement the method as described in any one of the first to third aspects.

[0020] Eighthly, this application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the method of any one of the first to third aspects.

[0021] Ninthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the method as described in any one of the first to third aspects.

[0022] In a tenth aspect, this application also provides a video quality assessment model, which includes: a motion distortion assessment module, a semantic distortion assessment module, and / or a semantic consistency assessment module;

[0023] The motion distortion assessment module generates motion distortion loss for the target video frame sequence of the input video quality assessment model. The motion distortion loss describes the degree to which the motion state of a preset object in the sample video frame sequence affects the quality of the sample video frame sequence. The semantic distortion assessment module generates semantic feature loss for the target video frame sequence. The semantic feature loss describes the degree to which the semantic discriminability of the sample video frame sequence affects the quality of the sample video frame sequence.

[0024] The semantic consistency evaluation module is used to generate semantic consistency loss for the target video frame sequence. The semantic consistency loss describes the degree of semantic alignment between the sample video frame sequence corresponding to the training sample group and the quality description text corresponding to the sample video frame sequence, and the degree of impact on the quality of the sample video frame sequence.

[0025] In the eleventh aspect, this application also provides a video-based semantic distortion assessment model, characterized in that it includes: a mask reconstruction unit and a feature processing unit;

[0026] The mask reconstruction unit is used to perform content reconstruction processing on the target video frame sequence to be evaluated for semantic distortion, so as to obtain the reconstructed video frame sequence corresponding to the target video frame sequence.

[0027] The feature processing unit is used to determine the semantic feature loss corresponding to the target video frame sequence based on the target video frame sequence and the reconstructed video frame sequence; the semantic feature loss is used to describe the semantic distortion of the target video frame sequence.

[0028] Compared with existing technologies, the training method of the video quality assessment model provided in the first aspect of this application has the following advantages:

[0029] The training method for the video quality assessment model provided in the first aspect of this application firstly obtains a training dataset; the training dataset includes at least one training sample group, and each training sample group includes at least: a sample video frame sequence and a set of descriptive text corresponding to the sample video frame sequence; the set of descriptive text includes at least one pre-defined video quality descriptive text used to represent high-quality semantics; the video quality assessment model to be trained is trained according to the training dataset using the following training process: a training sample group from the training dataset is input into the video quality assessment model to be trained, and the motion distortion assessment module, the semantic distortion assessment module, and / or the semantic consistency assessment module generate motion distortion loss, semantic feature loss, and / or semantic consistency loss corresponding to the training sample group, respectively; the model parameters of the video quality assessment model to be trained are adjusted according to the motion distortion loss, semantic feature loss, and / or semantic consistency loss corresponding to a preset number of training sample groups; the training process is repeated using a sufficient number of training sample groups from the training dataset until the preset end-of-training condition is met, thereby obtaining the trained video quality assessment model.

[0030] In the above training method, motion distortion evaluation module, semantic distortion evaluation module, and / or semantic consistency evaluation module are used to evaluate the motion distortion loss, semantic feature loss, and / or semantic consistency loss of the sample video frame sequence, respectively. The model parameters of the video quality evaluation model to be trained are adjusted based on these three types of losses, so that the trained video quality evaluation model can learn to evaluate the video frame sequence from multiple distortion dimensions (motion distortion, semantic distortion, and semantic consistency distortion), and can adaptively handle the quality evaluation of video frame sequences with complex distortion combinations, and the model has strong generalization ability.

[0031] Compared with the prior art, the video quality assessment method provided in the second aspect of this application has the following advantages:

[0032] The video quality assessment method provided in the second aspect of this application can use the video quality assessment model trained by the training method in the first aspect to assess the quality of the target video frame sequence to be assessed. The video quality assessment model can assess the video quality of the video frame sequence from multiple distortion dimensions (motion distortion, semantic distortion, semantic consistency distortion), making the quality score corresponding to the target video frame sequence more accurate.

[0033] Compared with existing technologies, the video quality assessment method provided in the third aspect of this application has the following advantages:

[0034] The video quality assessment method provided in the third aspect of this application performs content reconstruction processing on the target video frame sequence to be assessed, obtaining a reconstructed video frame sequence corresponding to the target video frame sequence. Based on the target video frame sequence and the reconstructed video frame sequence, the semantic feature loss corresponding to the target video frame sequence is determined. This semantic feature loss can be used to describe the degree to which the semantic discriminability of the sample video frame sequence affects the quality of the sample video frame sequence. Therefore, a target quality score corresponding to the target video frame sequence is determined based on the semantic feature loss. In this way, the information loss of the target video frame sequence at the semantic level can be directly measured, enabling effective utilization of semantic recognition priors when scoring the quality of the target video frame sequence, thereby improving the accuracy and realism of the quality score. Attached Figure Description

[0035] Figure 1 This is a flowchart illustrating an example of the training method for the video quality assessment model provided in this application;

[0036] Figure 2 This is a schematic diagram of an example structure of the video quality assessment model provided in an embodiment of this application;

[0037] Figure 3 This is a flowchart illustrating an example of the video quality assessment method provided in this application;

[0038] Figure 4 This is another example of a flowchart illustrating the video quality assessment method provided in this application;

[0039] Figure 5 This is a schematic diagram of an example of a training device for the video quality assessment model provided in this application;

[0040] Figure 6 This is a schematic diagram of an example of the video quality assessment device provided in this application;

[0041] Figure 7 This is a structural block diagram of the electronic device provided in this application. Detailed Implementation

[0042] To enable those skilled in the art to better understand the technical solutions of this application, the application will be clearly and completely described below with reference to the accompanying drawings of the embodiments. However, this application can be implemented in many other ways different from those described below. Therefore, based on the embodiments provided in this application, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this application.

[0043] It should be noted that the terms "first," "second," "third," etc., in the claims, specification, and drawings of this application are used to distinguish similar objects and are not used to describe a specific order or sequence. Such data are interchangeable where appropriate so that the embodiments of this application described herein can be implemented in a sequence other than that shown or described in this application. Furthermore, the terms "comprising," "having," and their variations are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products, or apparatuses.

[0044] It should be understood that in the embodiments of this application, "at least one" means one or more, and "more than one" means two or more. "And / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. The character " / " generally indicates that the related objects before and after it are in an "or" relationship. "Contains A, B and / or C" means containing any one, two, or three of A, B, and C.

[0045] It should be understood that in the embodiments of this application, "B corresponding to A", "B corresponding to A", "A corresponds to B" or "B corresponds to A" means that B is associated with A, and B can be determined based on A. Determining B based on A does not mean that B is determined solely based on A; B can also be determined based on A and / or other information.

[0046] To facilitate understanding of the various embodiments of this application, the relevant technical terms and technical means involved in this application will first be introduced.

[0047] 1. Video Masked Autoencoder (VideoMAE): A self-supervised learning model specifically designed for video data. This model reconstructs the masked portions by masking (i.e., occluding or hiding) certain frames or spatial regions in the input video and then using the unmasked portions of the frames or spatial regions.

[0048] 2. Time-Space Transformer (TimeSformer): A video understanding model based on the Transformer architecture, designed to effectively capture spatiotemporal information in video data. Unlike traditional convolutional neural networks, TimeSformer does not rely on convolutional operations to capture relationships between video frames. Instead, it directly models the spatiotemporal relationships in the video through a self-attention mechanism. Specifically, each frame in the video frame sequence is first divided into a series of non-overlapping patches. These patches contain both spatial and temporal information. Based on these patches, spatial attention is applied between different patches at a single point in time (i.e., within the same frame) to capture intra-frame spatial relationships. Temporal attention is applied across the temporal dimension (i.e., patches at the same position across different frames) to capture changes in actions or events over time. Combining these two approaches comprehensively considers the spatiotemporal dynamics in the video.

[0049] 3. The Pearson Linear Correlation Coefficient (PLCC) is an indicator used to measure the degree of linear correlation between two variables. Its value range includes -1, 0, and +1; where -1 indicates a perfect negative correlation; 0 indicates no linear correlation; and +1 indicates a perfect positive correlation. In this application, the PLCC index is used to test the linear correlation between the quality prediction score output by the trained video quality assessment model and the target quality score.

[0050] 4. Spearman's Rank Correlation Coefficient (SROCC) is an indicator used to measure the strength of a monotonic relationship between two variables. A monotonic relationship means that the two variables always trend in the same direction (if one increases, the other always increases; or if one increases, the other always decreases), but it is not necessarily linear. Its value range includes -1, 0, and +1; where +1 indicates a perfectly positive monotonic correlation (when one variable increases, the other variable also strictly increases); -1 indicates a perfectly negative monotonic correlation (when one variable increases, the other variable strictly decreases); and 0 indicates no monotonic correlation. In this application, the SROCC index is used to test the strength of the monotonic relationship between the predicted quality score output by the trained video quality assessment model and the target quality score.

[0051] To address the technical problems described above, embodiments of this application provide a training method, apparatus, electronic device, and computer-readable storage medium for a video quality assessment model. The aim is to solve or partially solve the aforementioned technical problems and train a video quality assessment model with strong generalization ability, suitable for evaluating the quality of videos with various complex distortion types.

[0052] The training method for the video quality assessment model provided in this application can be executed by an electronic device, which can be a terminal device, a server device, or a system combining both. The terminal can be a smartphone, tablet, laptop, or other similar device. The server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. It is understood that this application does not specifically limit the entity executing the training method for the video quality assessment model.

[0053] The technical solution of this application will be described in detail below through specific embodiments. It should be noted that the following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0054] like Figure 1 As shown, Figure 1 This is an example flowchart of a training method for a video quality assessment model provided in the first embodiment of this application. It should be noted that the steps shown may be executed in a different logical order than that shown in the flowchart. The method may include the following steps S110 to S130.

[0055] Step S110: Obtain the training dataset; the training dataset includes multiple training sample groups, each training sample group including at least: a sample video frame sequence and a set of descriptive text corresponding to the sample video frame sequence; the set of descriptive text includes at least one pre-defined video quality descriptive text used to represent high-quality semantics.

[0056] In this embodiment, the training dataset refers to the set of training samples used to train the video quality assessment model to be trained. That is, in this application, the video quality assessment model to be trained is trained using this training dataset to obtain the trained video quality assessment model, and the quality of video frame sequences is evaluated using this trained video quality assessment model. The video quality assessment model to be trained includes at least: a motion distortion assessment module, a semantic distortion assessment module, and / or a semantic consistency assessment module. This embodiment does not specifically limit the combination of the above three modules included in the video quality assessment model to be trained.

[0057] Reference Figure 2 , Figure 2 A schematic diagram of an example structure of the video quality assessment model provided in this application is provided. Figure 2 The provided video quality assessment model 200 includes a motion distortion assessment module 210, a semantic distortion assessment module 220, and a semantic consistency assessment module 230. For a detailed introduction to the motion distortion assessment module 210, the semantic distortion assessment module 220, and the semantic consistency assessment module 230, please refer to the following description, which will not be elaborated here.

[0058] Specifically, each training sample group may include a sequence of sample video frames with one or more different types of distortion, which may include, but are not limited to, one or more of the following: spatial structure deformation, compression distortion (blocking effect), temporal artifacts, preset object motion distortion, etc.

[0059] Furthermore, the aforementioned video quality description text is a pre-defined descriptive text representing high quality. For example, quality description text may include, but is not limited to: high-quality video, clear picture, sharp motion edges, no obvious noise, vibrant colors, rich details, and clear image quality.

[0060] After obtaining the above training dataset, the trained video quality assessment model is obtained through the following steps S120 to S130.

[0061] Step S120: Train the video quality evaluation model to be trained using the training dataset according to the training process shown in steps S121 to S122 below:

[0062] Step S121: Input a training sample group from the training dataset into the video quality assessment model to be trained. The motion distortion assessment module generates the motion distortion loss corresponding to the training sample group, the semantic distortion assessment module generates the semantic feature loss, and / or the semantic consistency assessment module generates the semantic consistency loss.

[0063] In this embodiment, the video quality assessment model to be trained generates motion distortion loss, semantic feature loss, or semantic consistency loss in the same way for each training sample group in the training dataset. Therefore, only the specific process of generating motion distortion loss, semantic feature loss, and semantic consistency loss for a training sample group corresponding to the video quality assessment model to be trained is introduced.

[0064] The aforementioned motion distortion loss is used to describe the degree of influence of the motion state of a preset object in the sample video frame sequence on the quality of the sample video frame sequence; that is, the aforementioned motion distortion evaluation module is used to evaluate the quality impact of the motion state of the preset object in the sample video frame sequence in the training sample group on the sample video frame sequence, with the aim of improving the motion smoothness and physical rationality of the video sequence in the time dimension.

[0065] In this context, the preset object in the sample video frame sequence refers to a visually distinct entity with independent semantic identity that appears in the pre-specified video frame sequence. This entity is typically a physical or virtual entity with clearly defined boundaries, shape, and behavior. For example, in a football match video, the football player, the football, the goal, and the overall atmosphere can all be considered independent objects. In this embodiment, the sample video frame sequence may include one or more objects; preferably, the preset object in this embodiment is the main object in the sample video frame sequence, i.e., the object of focus.

[0066] The motion state of the aforementioned preset object may include, but is not limited to, at least one of the following: motion stability, geometric deformation, motion blur, etc. Specifically, the motion stability of the preset object refers to whether the changes in the position, posture, velocity, and acceleration of the preset object relative to other objects in different video frames are smooth, consistent, and conform to physical laws or user expectations, without unexpected jitter, jumping, drifting, or flickering; the geometric deformation (i.e., distortion) of the preset object refers to the phenomenon that the preset object exhibits non-realistic shape changes such as geometric distortion, stretching, tilting, blurring, or breakage due to motion in different video frames, but when the geometric structure is deformed, the corresponding local texture may still be clear, which is a change in geometric structure; the motion blur of the preset object refers to the phenomenon that the preset object produces edge trailing, loss of detail, and blurred outline in the video frame sequence due to motion, but the overall position / shape remains unchanged.

[0067] Based on this, in an optional implementation, the training sample group also includes the real trajectory features of the preset object in the sample video frame sequence; when the motion state of the preset object in the video frame sequence includes the degree of motion stability, the motion distortion loss is specifically implemented through the following steps S1210a to S1212a.

[0068] Step S1210a: Input the sample video frame sequence from the training sample group into the motion distortion evaluation module. The motion distortion evaluation module determines the predicted motion trajectory features of the preset object in the sample video frame sequence based on the sample video frame sequence.

[0069] In this embodiment, the real trajectory features of the preset object in the sample video frame sequence are used to describe the precise sequence of the actual position of the preset object in the space given by the sample video frame over time; the real trajectory features are obtained by marking in the sample video frames in advance, and this application will not elaborate on the marking method.

[0070] The predicted motion trajectory features of the aforementioned preset object in the sample video frame sequence are used to describe the motion path of the preset object predicted by the motion distortion assessment module in the future time period based on the environment of the sample video frame sequence.

[0071] In an optional implementation, refer to Figure 2 The motion distortion assessment module 210 may include a target tracking unit 211 and a trajectory prediction unit 212. Based on this, in this optional embodiment, the predicted motion trajectory characteristics of the preset object can be determined through the following steps S1 to S4.

[0072] Step S1: The target tracking unit generates a current state feature sequence of the preset object in the current video frame based on the historical state feature sequence of the preset object in the historical video frames before the current video frame and the current state feature in the current video frame;

[0073] In this embodiment, the target tracking unit 211 may be constructed using a Transformer-based tracking encoder (TransTrack), which will not be described in detail in this embodiment.

[0074] It is understandable that in the model processing of the video frame sequence, each video frame in the sequence is sequentially input into the target tracking unit 211. Based on this, the aforementioned current video frame refers to the video frame input to the target tracking unit 211 at the current moment; the aforementioned historical video frame refers to the video frames that have been input into the target tracking unit 211 before the current moment.

[0075] In this embodiment, the state features of the preset object are used to describe the high-dimensional semantic vector extracted by the target tracking unit 211 for the preset object in the sample video frame sequence. The high-dimensional semantic vector can simultaneously encode the appearance information, spatial location (including position coordinates and scale), context information (relationship with other objects), motion direction and other state information of the preset object. In short, the state features are the feature vectors generated by the target tracking unit 211 when it detects the preset object in the sample video frame, and are used to uniquely identify the identity and state of the preset object.

[0076] The aforementioned current state feature refers to the high-dimensional semantic vector extracted by the target tracking unit 211 for the preset object in the current video frame; the aforementioned historical state feature sequence refers to the feature sequence composed of the high-dimensional semantic vectors extracted by the target tracking unit 21 for the preset object at each historical moment before the current moment.

[0077] That is, the purpose of step S1 is to fuse the current state features and historical state feature sequences corresponding to the preset object based on the global attention mechanism of Transformer to obtain the current state feature sequence corresponding to the current moment. In this way, the spatial position change and temporal motion pattern of the preset object at the current moment can be encoded.

[0078] After the target tracking unit 211 generates the current state feature sequence corresponding to the preset object, it is input into the trajectory prediction unit 212, and the following step S2 is executed.

[0079] Step S2: The trajectory prediction unit determines the predicted state features of the preset object in the next video frame after the current video frame based on the current state feature sequence. The predicted state features include the predicted position features of the preset object in the next video frame.

[0080] Specifically, in this embodiment, the trajectory prediction unit 212 may be a GRU (Gated Recurrent Unit) network. The spatial location prediction process of the GRU network for the preset object will not be described in this embodiment.

[0081] Furthermore, after determining the predicted state features of the preset object in the next video frame, it is detected whether the next video frame is the last video frame in the sample video frame sequence, and steps S3 or S4 are executed according to different detection results.

[0082] Step S3: When it is detected that the next video frame is not the last video frame in the sample video frame sequence, the next video frame is determined as the current video frame, and the process returns to execute S1 to S3.

[0083] Step S4: When the next video frame is detected to be the last video frame in the sample video frame sequence, the predicted motion trajectory features of the preset object are determined based on the predicted state features of the preset object in each video frame.

[0084] That is, for each video frame in the sample video frame sequence, a predicted state feature sequence corresponding to the preset object is generated iteratively, and the motion trajectory of the preset object over time is determined based on the predicted position features in each predicted state feature of the predicted state feature sequence.

[0085] After obtaining the predicted motion trajectory features of the preset object, the following steps S1211a to S1212a are executed to determine the motion distortion loss corresponding to the preset object.

[0086] Step S1211a: Determine the trajectory feature deviation between the predicted motion trajectory features of the preset object and the actual motion trajectory features of the preset object in the sample video frame sequence.

[0087] In an optional specific embodiment, for one of the predicted position features in the predicted motion trajectory features, the position feature deviation between the predicted position feature and the actual position feature corresponding to the predicted position feature in the actual motion trajectory can be calculated; and the mean square error can be calculated based on the position feature deviation corresponding to each predicted position feature in the predicted motion trajectory features, and the mean square error can be determined as the trajectory feature deviation.

[0088] Step S1212a: Determine the motion distortion loss of the preset object based on the trajectory feature deviation; the trajectory feature deviation is used to describe the motion stability of the preset object, and the trajectory feature deviation is negatively correlated with the motion stability.

[0089] In this embodiment, the trajectory feature deviation is negatively correlated with the degree of motion stability; that is, the higher the mean square error, the lower the motion stability of the preset object, the more unstable the motion of the preset object or the more severe the shaking, and the lower the mean square error, the more stable the motion of the preset object or the less severe the shaking.

[0090] In an optional implementation, when the motion state of the preset object includes motion stability, geometric deformation, and motion blur, the motion distortion loss is specifically achieved through steps S1210a to S1211a, and the following steps S1213a to S1215a. Steps S1210a to S1211a can be referred to the preceding description.

[0091] Step S1213a: The motion distortion assessment module determines the visual features corresponding to the preset object in at least two different video frames adjacent or near each other in the sample video frame sequence based on the sample video frame sequence.

[0092] Step S1214a: Determine the first similarity between the visual features corresponding to the preset object in different video frames; the first similarity is negatively correlated with the degree of geometric deformation or motion blur of the preset object in the video frame sequence.

[0093] Step S1215a: Determine the motion distortion loss of the preset object based on the trajectory feature deviation and the first similarity.

[0094] That is, after inputting the sample video frame sequence into the motion distortion assessment module, for the preset object, in addition to calculating the trajectory feature deviation between its corresponding predicted motion trajectory features and the real motion trajectory features, the module also calculates the first similarity between the visual features corresponding to the preset object in different video frames, so as to jointly determine the motion distortion loss of the preset object through the trajectory feature deviation and the first similarity.

[0095] As mentioned above, the target tracking unit encodes the appearance information of the preset object for the state features extracted in each video frame. That is, the encoded appearance information is the visual feature corresponding to the preset object.

[0096] The first similarity between the aforementioned visual features is specifically the mean cosine similarity of the visual features of the preset object across different video frames, which is used to measure the average similarity between the visual features.

[0097] In this embodiment, the first similarity is negatively correlated with the degree of geometric deformation or motion blur of the preset object in the video frame sequence. Specifically, the larger the first similarity, the less significant the change in visual features of the preset object between different frames, indicating that the preset object has low motion blur or small geometric deformation in the sample video frame sequence; conversely, the smaller the first similarity, the more significant the change in visual features of the preset object between different frames, meaning that the video is more likely to have motion blur or geometric deformation.

[0098] In this embodiment, the trajectory deviation and the first similarity can be fused to obtain the motion distortion loss of the preset object. That is, by combining the mean square error and the first similarity, the degree of quality degradation caused by motion distortion such as motion stability, geometric deformation, and motion blur of the sample video frame sequence can be evaluated in a multi-dimensional and accurate manner.

[0099] Next, we will introduce the relevant schemes for generating semantic feature loss corresponding to the training sample groups in the semantic distortion assessment module.

[0100] In this embodiment, the aforementioned semantic feature loss is used to describe the degree to which the semantic discriminability of the sample video frame sequence affects the quality of the sample video frame sequence; wherein, the semantic discriminability is used to describe whether the semantic information of the content in the video sequence is sufficiently clear, complete, and easily understood and distinguished by humans or machines. That is, the aforementioned semantic distortion evaluation module is used to evaluate the degree of semantic preservation and discriminability of the sample video frame sequences in the training sample group, so as to determine the quality impact on the sample video frame sequence through the degree of semantic preservation and discriminability.

[0101] In optional specific embodiments, refer to Figure 2 The semantic distortion evaluation module 220 includes at least a mask reconstruction unit 221 and a feature processing unit 222. Based on this, the semantic feature loss corresponding to the training sample group can be determined through the following steps S1210b to S1211b.

[0102] Step S1210b: Input the sample video frame sequence from the training sample group into the mask reconstruction unit, perform content reconstruction processing on the sample video frame sequence, and obtain the reconstructed video frame sequence corresponding to the sample video frame sequence.

[0103] In this embodiment, the mask reconstruction unit can specifically be a VideoMAE pre-trained on Kinetics-400. The pre-training process of this VideoMAE will not be described in detail. It should be noted that the pre-trained VideoMAE can infer the global structure and motion rules based on the locally visible image patches of the video frame sequence to reconstruct the completed video frame sequence.

[0104] In an optional specific implementation, the sample video frame sequence can be reconstructed through the following steps S10 to S30 to obtain the reconstructed video frame sequence corresponding to the sample video frame sequence.

[0105] Step S10: The mask reconstruction unit performs block processing on each video frame in the sample video frame sequence to obtain the image block set corresponding to the sample video frame sequence.

[0106] In this embodiment, segmenting each video frame into blocks means spatially dividing each video frame into image blocks. The size of each image block remains consistent, and the specific size is not specifically limited in this embodiment.

[0107] Step S20: Based on the preset mask rate, determine the first type of image blocks and the second type of image blocks in the image block set; the first type of image blocks are the image blocks to be masked, and the second type of image blocks are the image blocks that are not masked.

[0108] The aforementioned preset masking rate refers to the proportion of image blocks that are randomly or strategically masked out of the total number of image blocks in the set of image blocks corresponding to the input sample video frame sequence. For example, the preset masking rate can be any one of 70% to 90%, such as 80%, 75%, etc. For example, with a preset masking rate of 80%, 80% of the image blocks in the set of image blocks corresponding to the sample video frame sequence are randomly selected as the first type of image blocks for masking; the remaining 20% ​​are used as the second type of image blocks.

[0109] Furthermore, in this embodiment, the random seed used by the mask reconstruction unit to randomly select the first type of image patch is a fixed seed, to ensure that the same "random" occlusion mode is used for different sample video frame sequences. In addition, VideoMAE can intelligently filter out the key spatial regions (i.e., the spatial regions occupied by the moving subject (i.e., the preset object) in each video frame of the sample video frame sequence, and randomly select the image patches corresponding to most of the key spatial regions as the first type of image patch, so that VideoMAE can focus more on learning the most difficult-to-predict key regions such as the moving subject, i.e., the region where the preset object is located.

[0110] This ability to focus on core spatial features makes the masked reconstruction unit more sensitive to distortions in these key regions when reconstructing sample video frame sequences. Simultaneously, the high-density mask forces the model to infer the masked content from limited visible information, significantly enhancing the model's ability to perceive temporal distortions (such as inter-frame jitter and blur). When motion distortion exists in the video, the reconstruction error of the masked region increases significantly, directly reflecting the degree of semantic information loss.

[0111] Step S30: Mask the first type of image block, and based on the second type of image block, reconstruct the image at the position of the masked first type of image block in the corresponding video frame sequence to obtain the reconstructed video frame sequence corresponding to the sample video frame sequence.

[0112] In this embodiment, masking the first type of image blocks means obscuring (or hiding) them. After obscuring each first type of image block, image reconstruction is performed on the position of the masked first type of image block in the corresponding video frame sequence based on the unobscurated second type of image blocks. This means the mask reconstruction unit recovers the video frame sequence based on the local image blocks, and the recovered video frame sequence is the reconstructed video frame sequence. This reconstruction process will not be described in detail in this embodiment.

[0113] Step S1211b: The feature processing unit determines the semantic feature loss corresponding to the training sample group based on the sample video frame sequence and the reconstructed video frame sequence.

[0114] In this embodiment, the feature processing unit is specifically used to extract features and determine semantic representations for the sample video frame sequence and the reconstructed video frame sequence. Specifically, the feature processing unit may be a TimeSformer.

[0115] In an optional implementation, the semantic feature loss can be determined through the following steps S40 to S50.

[0116] Step S40: Determine the semantic reconstruction bias between the sample video frame sequence and the reconstructed video frame sequence; determine the classification stability of the preset object in each reconstructed video frame in the reconstructed video frame sequence; and / or, determine the video classification reconstruction bias between the sample video frame sequence and the reconstructed video frame sequence.

[0117] Regarding step S40, the specific steps for determining the aforementioned semantic reconstruction bias will be introduced first.

[0118] In this embodiment, the semantic reconstruction deviation is the semantic reconstruction deviation between the sample video frame sequence and the corresponding reconstructed video frame sequence. Based on this reconstruction deviation, it can be determined whether the reconstructed video frame sequence can effectively express the original semantics of the sample video frame sequence. If not, the reconstruction deviation can reflect that the sample video frame sequence has semantic quality loss, resulting in a large reconstruction deviation when reconstructing it.

[0119] Specifically, when the sample video frame sequence has quality defects such as blurring and mosaic, the mask reconstruction unit, when reconstructing the sample video frame sequence with quality defects such as blurring and mosaic, does not recognize the above-mentioned "unnatural" distortions such as blurring and mosaic, causing the reconstruction deviation to spike abnormally on the original basis. This abnormal increment can effectively reflect the damage to the semantic information of the sample video frame sequence. Therefore, the semantic feature loss corresponding to the training sample group can be determined by this semantic reconstruction deviation.

[0120] In an optional specific implementation, the above-mentioned semantic reconstruction deviation can be determined through the following steps S41 to S43.

[0121] Step S41: Extract the first semantic features corresponding to the sample video frame sequence, and extract the second semantic features corresponding to the reconstructed video frame sequence.

[0122] Step S42: Determine the second similarity between the first semantic feature and the second semantic feature.

[0123] Step S43: Determine the second similarity as semantic reconstruction bias.

[0124] In this embodiment, the second similarity is the cosine similarity between the first semantic feature and the second semantic feature, used to quantify the semantic deviation between the sample video frame sequence and the reconstructed video frame sequence. The specific calculation method for this second similarity will not be elaborated further.

[0125] Secondly, the steps for determining the stability of the above classifications will be introduced.

[0126] The classification stability of the aforementioned preset object in each reconstructed video frame in the reconstructed video frame sequence refers to the degree of fluctuation in the category confidence of the aforementioned preset object in each reconstructed video frame when classifying a specific subject in each video frame of the aforementioned reconstructed video frame sequence.

[0127] In this embodiment, based on the classification stability, it can be determined whether the fluctuation of the category confidence of the preset object in each reconstructed video frame of the reconstructed video frame sequence is stable. If it is unstable, it reflects that the quality of the reconstructed video frame sequence is poor, and further reflects that the sample video frame sequence has quality degradation. That is, due to the quality degradation of the sample video frame sequence (such as severe jitter, blur or noise), the visual performance of the preset object in each reconstructed video frame of the reconstructed video frame sequence is inconsistent, or even difficult to identify, resulting in drastic fluctuations in the classification of the preset object in each reconstructed video frame. If the above classification stability tends to be stable, it indicates that the quality of the sample video frame sequence is high, and the quality of its corresponding reconstructed video frame sequence is high.

[0128] In an optional implementation, the stability of the classification can be determined by the following steps S44 to S45.

[0129] Step S44: Determine the classification confidence of the preset object in each reconstructed video frame of the reconstructed video frame sequence.

[0130] In this embodiment, the classification confidence refers to the degree of certainty that the generated preset object belongs to the preset category when object classification prediction is performed for each reconstructed video frame.

[0131] In optional specific implementations, refer to Figure 2The semantic distortion assessment module 220 also includes a first classification model 223. Each reconstructed video frame in the reconstructed video frame sequence can be input into the first classification model 223. The first classification model 223 generates classification probabilities for each reconstructed video frame in the reconstructed video frame sequence. The classification probability represents the classification confidence of the first classification model for the classification result of the preset object in each reconstructed video frame. For example, assuming the preset object in the reconstructed video frame sequence (e.g., including 3 reconstructed video frames) is a cat, after inputting each reconstructed video frame into the first classification model 223, when the first classification model 223 performs classification prediction for the first reconstructed video frame, the classification confidence of the preset object belonging to the preset category "cat" is 0.9; when performing classification prediction for the second reconstructed video frame, the classification confidence of the preset object belonging to the preset category "cat" is 0.88; and when performing classification prediction for the third reconstructed video frame, the classification confidence of the preset object belonging to the preset category "cat" is 0.91.

[0132] After determining the classification confidence of the preset object in each reconstructed video frame, the following step S45 is performed.

[0133] Step S45: Determine the confidence variance of the preset object in the reconstructed video frame sequence based on the classification confidence of the preset object in each reconstructed video frame sequence; the confidence variance is used to describe the classification stability of the preset object in the reconstructed video frame sequence.

[0134] In this embodiment, the confidence variance is used to measure the fluctuation of the classification confidence of each frame corresponding to the preset object, that is, to measure the stability of the object classification in the reconstructed video frame sequence. If the confidence variance is large, the classification confidence of the preset object in the reconstructed video frame sequence of the first classification model fluctuates greatly, which can reflect that the visual observability of the preset object fluctuates greatly in different frames of the reconstructed video frame sequence, and thus reflect that the quality of the reconstructed video frame sequence is poor, such as severe jitter, blur or noise, which makes the appearance of the preset object inconsistent in different frames, and may even be difficult to identify; while if the confidence variance is small, the classification confidence of the preset object in the reconstructed video frame sequence fluctuates less, which can reflect that the visual observability of the preset object fluctuates less in different reconstructed video frames of the reconstructed video frame sequence, and thus indirectly reflect that the quality of the reconstructed video frame sequence is high.

[0135] Continuing with the example above, suppose the reconstructed video frame quality is very poor, such as having severe jitter, blur, or noise, causing the subject "cat" to look different in different reconstructed video frames, or even be difficult to identify at times. In this case, the confidence score of the first classification for the subject belonging to the "cat" category will fluctuate drastically. For example, the classification confidence score for the first frame is 0.9, the classification confidence score for the second frame becomes 0.5 due to blur, and the classification confidence score for the third frame becomes 0.7 due to jitter. This demonstrates that the degradation of video semantic quality can lead to changes in the confidence variance. Therefore, in this embodiment, the semantic quality loss of the sample video frame sequence is quantified by this confidence variance.

[0136] Next, we will introduce the specific implementation method for determining the video classification and reconstruction deviation between the sample video frame sequence and the reconstructed video frame sequence in step S40 above.

[0137] The aforementioned video classification reconstruction bias refers to the reconstruction bias between the sample video frame sequence and the corresponding reconstructed video frame sequence at the video classification level. Based on this reconstruction bias, it can be determined whether the video category to which the reconstructed video frame sequence belongs is consistent with the original category to which the sample video frame sequence belongs. If they are inconsistent, the video classification reconstruction bias can reflect that the sample video frame sequence has semantic quality loss, resulting in a large video classification reconstruction bias when it is reconstructed.

[0138] Specifically, when the sample video frame sequence has quality defects such as blurring and mosaic, the mask reconstruction unit does not recognize the "unnatural" distortions such as blurring and mosaic when it reconstructs the sample video frame sequence. This leads to incorrect reconstruction of the video content, which in turn causes prediction errors when predicting the video category for the reconstructed video frame sequence. This effectively reflects the damage to the semantic information of the sample video frame sequence.

[0139] In optional specific implementations, refer to Figure 2 The semantic distortion assessment module 220 also includes a second classification model 224, which can input the sample video frame sequence and the reconstructed video frame sequence into the second classification model 224 respectively. The second classification model 224 generates a first classification probability distribution and a second classification probability distribution for the sample video frame sequence and the reconstructed video frame sequence respectively. The first classification probability distribution includes the probability value of at least one video category to which the sample video frame sequence belongs. The second classification probability distribution includes the probability value of at least one video category to which the reconstructed sample video frame sequence belongs. The distribution difference between the first classification probability distribution and the second classification probability distribution is determined, and the distribution difference is determined as the video classification reconstruction bias.

[0140] Specifically, the distribution difference between the first and second classification probability distributions can be calculated using the Kullback-Leibler Divergence (KL divergence). The specific calculation method will not be described in detail in this embodiment.

[0141] For example, if the sample video frame sequence is a football match video with serious quality defects such as blurriness and pixelation, and the mask reconstruction unit reconstructs the football as a basketball, then the second classification model's output probability distribution for classifying the sample video frame sequence is [basketball: 0.1, football: 0.85; volleyball: 0.05]; while its output probability distribution for classifying the reconstructed video frame sequence is [basketball: 0.75; football: 0.2; volleyball: 0.05]. Thus, it can be seen that the significant difference between the first and second probability distributions reflects a poor reconstruction effect on the sample video frame sequence, and consequently, a poor quality of the sample video frame sequence.

[0142] Step S50: Determine the semantic feature loss corresponding to the training sample group based on the semantic reconstruction bias, classification stability, and / or video classification reconstruction bias; wherein the semantic reconstruction bias, classification stability, and video classification reconstruction bias are positively correlated with the semantic feature loss.

[0143] In this embodiment, the dimension used to determine the semantic feature loss corresponding to the training sample group can be selected according to actual needs. For example, when selecting to determine the semantic reconstruction deviation between the sample video frame sequence and the reconstructed video frame sequence, the semantic reconstruction deviation is determined as the semantic feature loss corresponding to the training sample group. When selecting to determine the classification stability of the preset object in each reconstructed video frame in the reconstructed video frame sequence, the confidence variance is determined as the semantic feature loss corresponding to the training sample group. Alternatively, when selecting to determine the video classification reconstruction deviation between the sample video frame sequence and the reconstructed video frame sequence, the video classification reconstruction deviation is determined as the semantic feature loss corresponding to the training sample group.

[0144] Alternatively, when simultaneously selecting and determining semantic reconstruction bias, classification stability, and / or video classification reconstruction bias, the semantic reconstruction bias, confidence variance, and video classification reconstruction bias can be fused to obtain fused features; and the fused features can be determined as the semantic feature loss corresponding to the training sample group.

[0145] Therefore, in this embodiment, when the sample video frame sequence has motion distortion or spatial distortion, the second type of image block may be unrecognizable when the mask reconstruction unit uses the unmasked second type of image block for reconstruction. This leads to a deviation in the semantic features of the reconstructed video frame sequence, resulting in a weaker classification stability of the preset object in each reconstructed video frame and a reconstruction deviation in video classification. Therefore, this embodiment can judge the semantic information loss of the sample video frame sequence based on multiple dimensions such as the semantic reconstruction deviation, classification stability, and video classification reconstruction deviation, making the determination of the semantic information loss more accurate.

[0146] Next, we will introduce the relevant schemes for generating semantic consistency loss corresponding to the training sample groups in the semantic consistency evaluation module.

[0147] The aforementioned semantic consistency loss describes the degree of semantic alignment between the sample video frame sequence corresponding to the training sample group and the quality descriptive text in the corresponding descriptive text set, and the degree of impact on the quality of the sample video frame sequence. That is, the semantic consistency evaluation module evaluates the degree of semantic alignment between the sample video frame sequence in the training sample group and the quality descriptive text in the corresponding descriptive text set, so as to determine the quality impact on the sample video frame sequence based on this semantic alignment.

[0148] In an optional specific embodiment, the semantic feature loss corresponding to the training sample group can be determined by the following steps S1210c to S1211c.

[0149] Step S1210c: Input the training sample group into the semantic consistency evaluation module. The semantic consistency evaluation module determines the visual semantic features of the sample video frame sequence and the textual semantic features of the quality descriptive text in the descriptive text set.

[0150] Optionally, the semantic distortion evaluation module 220 can specifically be a Contrastive Language-Image Pre-Training (CLIP) model. The CLIP model can specifically use the ViT-B / 32 architecture, that is, a Base-scale Vision Transformer (ViT) structure with a total of 12 layers, and the sample video frames in the input sample video frame sequence are sequentially divided into 32×32 pixel image blocks. The model structure and pre-training process of the CLIP model will not be described in detail.

[0151] Reference Figure 2The semantic distortion evaluation module 220 (CLIP model) includes at least a visual semantic feature extraction unit 231 and a text semantic feature extraction unit 232. After the training sample group is input into the semantic distortion evaluation module 220, the visual semantic feature extraction unit 231 extracts the visual semantic features of the sample video frame sequence in the training sample group, and the text semantic feature extraction unit 232 extracts the text semantic features of the quality description text in the above-mentioned description text set.

[0152] Step S1211c: Determine the semantic consistency loss corresponding to the training sample group based on visual semantic features and text semantic features.

[0153] In an optional implementation, a third similarity is calculated between visual semantic features and textual semantic features; this third similarity is defined as the semantic consistency loss corresponding to the training sample group; the third similarity is negatively correlated with the semantic consistency loss. Specifically, the third similarity is the cosine similarity between visual semantic features and textual semantic features. A higher cosine similarity indicates a lower semantic consistency loss for the sample video frame sequence, signifying a better match between the video content and the high-quality semantic description, i.e., a higher quality sample video frame sequence. Conversely, a lower cosine similarity indicates a higher semantic consistency loss for the sample video frame sequence, suggesting that the video content deviates from the high-quality semantic description and may exhibit severe visual or semantic distortion.

[0154] Reference Figure 2 The semantic distortion evaluation module 220 also includes a similarity calculation unit 233. After the visual semantic feature extraction unit 231 extracts the visual semantic features and the text semantic feature extraction unit 232 extracts the text semantic features, both the visual semantic features and the text semantic features are input into the similarity calculation unit 233, so that the similarity calculation unit 233 calculates the cosine similarity between the visual semantic features and the text semantic features. The cosine similarity is the semantic consistency loss corresponding to the input training sample group.

[0155] In this way, by calculating the cosine similarity between visual semantic features and textual semantic features, a comprehensive semantic matching measurement can be performed on the sample video frame sequence, thereby achieving a refined evaluation of the sample video frame sequence. This allows the video quality evaluation model to learn to understand and evaluate video quality from a human cognitive perspective, rather than simply staying at the pixel level or low-level features.

[0156] After calculating the motion distortion loss, semantic feature loss, and / or semantic consistency loss corresponding to a preset number of training sample groups according to the above implementation methods, the model parameters of the video quality assessment model to be trained are adjusted by performing the following step S122.

[0157] Step S122: Adjust the model parameters of the video quality assessment model to be trained according to the motion distortion loss, semantic feature loss and / or semantic consistency loss corresponding to the preset number of training sample groups.

[0158] In this embodiment, during the adjustment of the model parameters of the video quality assessment model to be trained, methods such as batch gradient descent, stochastic gradient descent, or mini-batch gradient descent can be used. Specifically, batch gradient descent means that the number of samples used for each parameter update is the entire training dataset; stochastic gradient descent means that the model parameters of the video quality assessment model to be trained are updated once when the motion distortion loss, semantic feature loss, and / or semantic consistency loss corresponding to a training sample group are calculated; mini-batch gradient descent means that when the motion distortion loss, semantic feature loss, and / or semantic consistency loss corresponding to a batch of training sample groups (not the entire training dataset) are calculated, the model parameters of the video quality assessment model to be trained are updated once based on the average loss corresponding to that batch of training sample groups. That is, the above-mentioned preset number can be 1, n, or N; where N is the total number of training sample groups in the training dataset, n is the preset batch size of the training sample groups, and n is less than N.

[0159] Preferably, this application uses mini-batch gradient descent to update the model parameters of the video quality assessment model to be trained.

[0160] In this embodiment, during the training process of the video quality assessment model to be trained, the motion distortion loss is intended to penalize motion anomalies (such as jitter, jumps, blur, drift, etc.) in the quality prediction results, so that the video quality assessment model to be trained can learn temporal smoothness and consistency (avoiding jitter and jumps), physical rationality of motion (such as continuous inertia and acceleration), and the ability to perceive real motion patterns (such as natural human movements and stable object trajectories).

[0161] The aforementioned semantic feature loss aims to penalize the behavior of "the reconstructed video frame sequence corresponding to the sample video frame sequence being inconsistent with the sample video frame sequence in terms of high-level semantic structure", so that the video quality assessment model to be trained learns an image representation that is more sensitive to human visual perception and has semantic fidelity, rather than just pixel-level approximation.

[0162] The aforementioned semantic consistency loss is designed to penalize inconsistencies in the representations of different modalities (i.e., video and text) in the shared semantic space, thereby forcing the video quality assessment model to learn a joint representation of semantic alignment across multiple modalities.

[0163] In an optional implementation, the training sample set further includes a target quality score corresponding to the sample video frame sequence; this target quality score can be a human cognitive score set for the sample video frame sequence. Based on this, in this optional implementation, the model parameters of the video quality assessment model to be trained can be adjusted through the following steps S1220 to S1221.

[0164] Step S1220: Based on the motion distortion loss, semantic feature loss, and / or semantic consistency loss corresponding to the preset number of training sample groups, output the quality prediction score corresponding to each training sample group.

[0165] In an optional implementation, refer to Figure 2 The video quality assessment model 200 to be trained also includes a dynamic fusion module 240 and a quality prediction network 250. Based on this, motion distortion loss, semantic feature loss, and / or semantic consistency loss can be input into the dynamic fusion module 240. The dynamic fusion module 240 generates a first weight for motion distortion loss, a second weight for semantic feature loss, and / or a third weight for semantic consistency loss. Then, the dynamic fusion module 240 performs weighted processing on the motion distortion loss, semantic feature loss, and / or semantic consistency loss according to the first weight, the second weight, and / or the third weight to obtain fusion loss features. Furthermore, the fusion loss features can be input into the quality prediction network 250. The quality prediction network 250 performs dimensionality compression processing on the fusion loss features and outputs the quality prediction score corresponding to the training sample group. The specific process of this dimensionality compression processing will not be described in detail in this embodiment.

[0166] The first weight is used to describe the significance of motion distortion loss in the sample video frame sequence; the second weight is used to describe the significance of semantic feature loss in the sample video frame sequence; and the third weight is used to describe the significance of semantic consistency loss in the sample video frame sequence.

[0167] In this embodiment, the dynamic fusion module 240 specifically adopts a learnable gating mechanism for dynamic fusion. The dynamic fusion module 240 can be a small neural network (such as a fully connected layer with a Sigmoid activation function). The learnable gating mechanism will not be described in detail in this embodiment.

[0168] In this embodiment, quality losses of one or more dimensions, such as motion distortion loss, semantic feature loss, and semantic consistency loss, can be considered to generate quality prediction scores corresponding to the training sample group. That is, this video quality assessment model is applicable to various complex distortion scenarios, achieving more accurate and robust video quality assessment. Furthermore, through the aforementioned dynamic fusion module 240, a set of dynamic weights can be adaptively generated based on motion distortion loss, semantic feature loss, and / or semantic consistency loss, so that each weight can reflect the significance of different types of distortion (motion distortion, semantic degradation, cognitive inconsistency) in the current sample video frame. For example, when the sample video frame sequence has severe jitter, the dynamic fusion module 240 assigns a higher first weight to the motion distortion loss; when the video semantic information of the sample video frame sequence is severely damaged, the dynamic fusion module 240 increases the second weight corresponding to the semantic feature loss; and when the video content deviates significantly from human cognitive expectations, the dynamic fusion module 240 increases the third weight corresponding to the semantic consistency loss. That is, the dynamic fusion module 240 can perform weight allocation based on data-driven principles, adaptively handle complex distortion combination scenarios, and enable the video quality assessment model to be trained to learn how to assess different types of video quality, thereby improving the generalization ability of the video quality assessment model.

[0169] Step S1221: Based on the principle of reducing the difference between the predicted quality score and the target quality score corresponding to each training sample group, adjust the model parameters of the video quality assessment model to be trained.

[0170] The principle is to reduce the difference between the predicted quality score and the target quality score for each training sample group, which is to make the average loss value corresponding to the preset number of training sample groups less than a preset threshold.

[0171] In a specific implementation, for one training sample group in a preset number of training sample groups, the quality prediction score corresponding to that training sample group is compared with the given target quality score, and the quality loss value corresponding to that training sample group is calculated. After calculating the quality loss value corresponding to each training sample group, the average loss value corresponding to the preset number of training sample groups is calculated. Based on the principle of making the average loss value less than a preset threshold, the model parameters of the video quality assessment model to be trained are adjusted.

[0172] Specifically, during the model parameter update phase of the video quality assessment model to be trained, its pre-set learning rate (e.g., 3×10⁻⁶) is invoked. -5 The algorithm updates the model parameters of the video quality assessment model to be trained using a gradient optimizer (such as AdamW) with weight decay (e.g., 0.01). The specific process will not be described in detail in this embodiment.

[0173] Step S130: Repeat the above training process using a sufficient number of training sample groups from the training dataset until the preset end-of-training conditions are met, and obtain the trained video quality evaluation model.

[0174] That is, in this embodiment, after updating the model parameters of the video quality assessment model to be trained once using a preset number of training sample groups, the next batch of preset number of training sample groups is input into the video quality assessment model to be trained, and the above training process is repeated until the preset end training conditions are met.

[0175] The preset conditions for ending training may include, but are not limited to: the loss value of the loss function of the video quality assessment model to be trained is less than or equal to a preset threshold; the number of training iterations reaches a preset number; the quality prediction score output by the video quality assessment model to be trained reaches a preset score; and the performance of the video quality assessment model to be trained on the training dataset no longer changes.

[0176] This concludes the introduction to the training method for the video quality assessment model to be trained.

[0177] The following section presents experimental data for performance evaluation of the trained video quality assessment model. Specifically, this application tested the trained video quality assessment model on the KoNViD-1k dataset, the YouTube-UGC dataset, the LIVE-VQC dataset, and the Kwai UGC dataset. The trained video quality assessment model demonstrated excellent performance in both PLCC and SROCC metrics on the aforementioned datasets. As shown in Table 1, Table 1 provides performance examples of the trained video quality assessment model in terms of PLCC and SROCC metrics on different datasets.

[0178] Table 1

[0179]

[0180] As shown in Table 1, when the KoNViD-1k dataset was used to perform performance testing on the trained video quality assessment model, the SROCC (Significant Response Rate of Correction) value between the predicted quality score and the target quality score was 0.910, and the PLCC (Proportional Response Rate of Correction) value was 0.915. When the YouTube-UGC dataset was used to perform performance testing on the trained video quality assessment model, the SROCC value between the predicted quality score and the target quality score was 0.893, and the PLCC value was 0.879. When the LIVE-VQC dataset was used to perform performance testing on the trained video quality assessment model, the SROCC value between the predicted quality score and the target quality score was 0.893, and the PLCC value was 0.879. The SROCC (Significant Response Rate) and PLCC (Plus Point Rate) values ​​between the predicted quality score and the target quality score output by the quality assessment model are 0.856 and 0.883, respectively. When the trained video quality assessment model is tested on the Kwai UGC dataset, the SROCC value between the predicted quality score and the target quality score is 0.844 and the PLCC value is 0.857. The corresponding SROCC and PLCC values ​​on each dataset are all above 0.8, indicating that the quality prediction scores output by the trained video quality assessment model on the KoNViD-1k, YouTube-UGC, LIVE-VQC, and Kwai UGC datasets are very close to the given target quality scores. The model has excellent performance, strong generalization ability, and adaptability to various types of video frame sequences.

[0181] The second embodiment of this application also provides a video quality assessment method corresponding to the training method embodiment of the video quality assessment model provided in the first embodiment. Since the video quality assessment method embodiment is basically similar to the training method embodiment of the video quality assessment model, it is described simply. For details of the relevant technical features and the effects achieved, please refer to the corresponding descriptions of the training method embodiment of the video quality assessment model provided above.

[0182] Reference Figure 3 The video quality assessment method may include the following steps S310 to S320.

[0183] Step S310: Obtain the target video frame sequence to be quality evaluated;

[0184] Step S320: Input the target video frame sequence into the video quality assessment model so that the video quality assessment model outputs the quality score corresponding to the target video frame sequence.

[0185] The video quality assessment model is trained using any of the training methods provided in the first embodiment. That is, the video quality assessment method is the practical application of the trained video quality assessment model. The target video frame sequence is the video segment to be evaluated.

[0186] The third embodiment of this application also provides a video quality assessment method. Since the details and effects of this video quality assessment method are similar to those in the training method of the video quality assessment model in the first embodiment, the similar parts will not be described further here. (Refer to...) Figure 4 The video quality assessment method may include the following steps S410 to S430.

[0187] Step S410: Perform content reconstruction processing on the target video frame sequence to be quality assessed to obtain the reconstructed video frame sequence corresponding to the target video frame sequence.

[0188] In an optional specific embodiment, the mask reconstruction unit described above can be used to perform content reconstruction processing on the target video frame sequence, which will not be elaborated here.

[0189] Step S420: Based on the target video frame sequence and the reconstructed video frame sequence, determine the semantic feature loss corresponding to the target video frame sequence; the semantic feature loss is used to describe the degree to which the semantic discriminability of the sample video frame sequence affects the quality of the sample video frame sequence.

[0190] In an optional implementation, the semantic feature loss corresponding to the target video frame sequence can be determined as follows: determine the semantic reconstruction deviation between the target video frame sequence and the reconstructed video frame sequence; determine the classification stability of the preset object in the target video frame sequence in each reconstructed video frame in the reconstructed video frame sequence; and / or, determine the video classification reconstruction deviation between the target video frame sequence and the reconstructed video frame sequence; and then determine the semantic feature loss corresponding to the training sample group based on the semantic reconstruction deviation, classification stability, and / or video classification reconstruction deviation; wherein, the semantic reconstruction deviation, classification stability, and video classification reconstruction deviation are positively correlated with the semantic feature loss, and the classification stability is negatively correlated with the semantic feature loss.

[0191] Step S430: Determine the target quality score corresponding to the target video frame sequence based on semantic feature loss.

[0192] In an optional implementation, motion distortion loss corresponding to the target video frame sequence can also be determined; motion distortion loss is used to describe the degree of influence of the motion state of the preset object in the sample video frame sequence on the quality of the sample video frame sequence; then, the target quality score corresponding to the target video frame sequence is determined based on motion distortion loss and / or semantic feature loss.

[0193] In an optional implementation, the quality description text corresponding to the target video frame sequence can also be obtained; based on the quality description text and the target video frame sequence, the semantic consistency loss corresponding to the target video frame sequence is determined; the semantic consistency loss is used to describe the degree of influence of the semantic alignment between the sample video frame sequence corresponding to the training sample group and the quality description text corresponding to the sample video frame sequence on the quality of the sample video frame sequence; and then, based on the motion distortion loss, semantic feature loss and / or semantic consistency loss, the target quality score corresponding to the target video frame sequence is determined.

[0194] Specifically, the target quality score corresponding to the target video frame sequence can be determined as follows based on motion distortion loss, semantic feature loss, and / or semantic consistency loss: determine the first weight corresponding to motion distortion loss, the second weight corresponding to semantic feature loss, and / or the third weight corresponding to semantic consistency loss; perform weighted processing on motion distortion loss, semantic feature loss, and / or semantic consistency loss according to the first weight, the second weight, and / or the third weight to obtain fusion loss features; determine the target quality score corresponding to the target video frame sequence based on the fusion loss features.

[0195] The fourth embodiment of this application also provides a training apparatus for a video quality assessment model corresponding to the training method embodiment for the video quality assessment model provided in the first embodiment. Since the apparatus embodiment is basically similar to the method embodiment, it is described simply. For details of the relevant technical features and the effects achieved, please refer to the corresponding descriptions of the training method embodiment for the video quality assessment model provided above.

[0196] like Figure 5 As shown, the training device 500 for the video quality assessment model provided in this embodiment includes: an acquisition unit 501 and a training unit 502; wherein, the video quality assessment model to be trained includes at least: a motion distortion assessment module, a semantic distortion assessment module, and / or a semantic consistency assessment module.

[0197] The acquisition unit 501 is used to acquire the training dataset; the training dataset includes at least one training sample group, and each training sample group includes at least: a sample video frame sequence and a set of descriptive text corresponding to the sample video frame sequence; the set of descriptive text includes at least one pre-set video quality descriptive text used to represent high-quality semantics;

[0198] Training unit 502 is used to train the video quality assessment model to be trained based on the training dataset using the following training process: A training sample group from the training dataset is input into the video quality assessment model to be trained; the motion distortion assessment module generates the motion distortion loss corresponding to the training sample group; the semantic distortion assessment module generates the semantic feature loss; and / or the semantic consistency assessment module generates the semantic consistency loss. The motion distortion loss describes the degree of influence of the motion state of a preset object in the sample video frame sequence on the quality of the sample video frame sequence; the semantic feature loss describes the degree of influence of the semantic discriminability of the sample video frame sequence on the quality of the sample video frame sequence; and the semantic consistency loss describes the degree of influence of the semantic alignment between the sample video frame sequence corresponding to the training sample group and the quality description text corresponding to the sample video frame sequence on the quality of the sample video frame sequence. The model parameters of the video quality assessment model to be trained are adjusted according to the motion distortion loss, semantic feature loss, and / or semantic consistency loss corresponding to a preset number of training sample groups. The training process is repeated using a sufficient number of training sample groups from the training dataset until the preset end-of-training condition is met, resulting in the trained video quality assessment model.

[0199] Optionally, the training sample group also includes the true trajectory features of the preset object in the sample video frame sequence; the training unit 502 is specifically used to input the sample video frame sequence in the training sample group into the motion distortion evaluation module. The motion distortion evaluation module determines the predicted motion trajectory features of the preset object in the sample video frame sequence based on the sample video frame sequence; determines the trajectory feature deviation between the predicted motion trajectory features of the preset object and the true motion trajectory features of the preset object in the sample video frame sequence; and determines the motion distortion loss of the preset object based on the trajectory feature deviation. The trajectory feature deviation is used to describe the motion stability of the preset object, and the trajectory feature deviation is negatively correlated with the motion stability.

[0200] Optionally, the training unit 502 is further configured to: determine, based on the sample video frame sequence, the visual features corresponding to the preset object in at least two adjacent or nearby different video frames in the sample video frame sequence; determine the first similarity between the visual features corresponding to the preset object in different video frames; the first similarity is negatively correlated with the degree of geometric deformation or motion blur of the preset object in the video frame sequence; and determine the motion distortion loss of the preset object based on the trajectory feature deviation and the first similarity.

[0201] Optionally, training unit 502 is also specifically used to fuse trajectory deviation and first similarity to obtain motion distortion loss of preset object.

[0202] Optionally, the motion distortion assessment module includes a target tracking unit and a trajectory prediction unit; the training unit 502 is also specifically used to perform the following steps S1 to S4:

[0203] S1: The target tracking unit generates a current state feature sequence corresponding to the preset object based on the historical state feature sequence of the preset object in the historical video frames before the current video frame and the current state feature in the current video frame;

[0204] S2: The trajectory prediction unit determines the predicted state features of the preset object in the next video frame after the current video frame based on the current state feature sequence. The predicted state features include the predicted position features of the preset object in the next video frame.

[0205] S3: When it is detected that the next video frame is not the last video frame in the sample video frame sequence, the next video frame is determined as the current video frame, and the execution of S1 to S3 is returned.

[0206] S4: When the next video frame is detected to be the last video frame in the sample video frame sequence, the predicted motion trajectory features of the preset object are determined based on the predicted position features of the preset object in each video frame.

[0207] Optionally, the training unit 502 is specifically used to calculate the position feature deviation between the predicted position feature and the corresponding real position feature in the real motion trajectory for a predicted position feature in the predicted motion trajectory features; calculate the mean square error based on the position feature deviation corresponding to each predicted position feature in the predicted motion trajectory features, and determine the mean square error as the trajectory feature deviation.

[0208] Optionally, the semantic distortion evaluation module includes a mask reconstruction unit and a feature processing unit; the training unit 502 is specifically used to input the sample video frame sequence from the training sample group into the mask reconstruction unit, perform content reconstruction processing on the sample video frame sequence, and obtain the reconstructed video frame sequence corresponding to the sample video frame sequence; the feature processing unit determines the semantic feature loss corresponding to the training sample group based on the sample video frame sequence and the reconstructed video frame sequence.

[0209] Optionally, the training unit 502 is specifically used by the mask reconstruction unit to perform block processing on each video frame in the sample video frame sequence to obtain a set of image blocks corresponding to the sample video frame sequence; according to a preset mask rate, determine the first type of image blocks and the second type of image blocks in the set of image blocks; the first type of image blocks are the image blocks to be masked, and the second type of image blocks are the image blocks that are not masked; perform mask processing on the first type of image blocks, and according to the second type of image blocks, perform image reconstruction on the position of the masked first type of image blocks in the corresponding video frame sequence to obtain the reconstructed video frame sequence corresponding to the sample video frame sequence.

[0210] Optionally, training unit 502 is specifically used to determine the semantic reconstruction bias between the sample video frame sequence and the reconstructed video frame sequence; determine the classification stability of the preset object in each reconstructed video frame in the reconstructed video frame sequence; and / or determine the video classification reconstruction bias between the sample video frame sequence and the reconstructed video frame sequence; and determine the semantic feature loss corresponding to the training sample group based on the semantic reconstruction bias, classification stability, and / or video classification reconstruction bias; wherein the semantic reconstruction bias and video classification reconstruction bias are positively correlated with the semantic feature loss, and the classification stability is negatively correlated with the semantic feature loss.

[0211] Optionally, the training unit 502 is specifically used to extract the first semantic features corresponding to the sample video frame sequence and extract the second semantic features corresponding to the reconstructed video frame sequence; determine the second similarity between the first semantic features and the second semantic features; and determine the second similarity as the semantic reconstruction bias.

[0212] Optionally, training unit 502 is specifically used to determine the classification confidence of the preset object in each reconstructed video frame of the reconstructed video frame sequence; and to determine the confidence variance of the preset object in the reconstructed video frame sequence based on the classification confidence of the preset object in each reconstructed video frame sequence; the confidence variance is used to describe the classification stability of the preset object in the reconstructed video frame sequence.

[0213] Optionally, the training unit 502 is specifically used to input each reconstructed video frame in the reconstructed video frame sequence into the first classification model, and the first classification model generates classification probabilities for each reconstructed video frame; the classification probability is used to represent the classification confidence of the first classification model for the classification result of the preset object in each reconstructed video frame.

[0214] Optionally, the training unit 502 is specifically used to input the sample video frame sequence and the reconstructed video frame sequence into the second classification model. The second classification model generates a first classification probability distribution and a second classification probability distribution for the sample video frame sequence and the reconstructed video frame sequence, respectively. The first classification probability distribution includes the probability value of at least one video category to which the sample video frame sequence belongs. The second classification probability distribution includes the probability value of at least one video category to which the reconstructed sample video frame sequence belongs. The distribution difference between the first classification probability distribution and the second classification probability distribution is determined, and the distribution difference is determined as the video classification reconstruction bias.

[0215] Optionally, the training unit 502 is specifically used to determine the semantic reconstruction bias as the semantic feature loss corresponding to the training sample group; to determine the classification stability as the semantic feature loss corresponding to the training sample group; or to determine the video classification reconstruction bias as the semantic feature loss corresponding to the training sample group.

[0216] Optionally, training unit 502 is specifically used to fuse semantic reconstruction bias, classification stability, and video classification reconstruction bias to obtain fused features; the fused features are determined as the semantic feature loss corresponding to the training sample group.

[0217] Optionally, the training unit 502 is specifically used to input the training sample group into the semantic consistency evaluation module. The semantic consistency evaluation module determines the visual semantic features of the sample video frame sequence and the textual semantic features of the quality descriptive text in the descriptive text set. Based on the visual semantic features and the textual semantic features, the semantic consistency loss corresponding to the training sample group is determined.

[0218] Optionally, training unit 502 is specifically used to calculate the third similarity between visual semantic features and textual semantic features; the third similarity is determined as the semantic consistency loss corresponding to the training sample group; the third similarity is negatively correlated with the semantic consistency loss.

[0219] Optionally, the training sample group also includes the target quality score corresponding to the sample video frame sequence; the training unit 502 is specifically used to output the quality prediction score corresponding to each training sample group according to the motion distortion loss, semantic feature loss and / or semantic consistency loss corresponding to the preset number of training sample groups; and to adjust the model parameters of the video quality assessment model to be trained based on the principle of reducing the difference between the quality prediction score and the corresponding target quality score of each training sample group.

[0220] Optionally, the video quality assessment model to be trained may also include a dynamic fusion module;

[0221] Training unit 502 is specifically used to input motion distortion loss, semantic feature loss, and / or semantic consistency loss into the dynamic fusion module. The dynamic fusion module generates a first weight for motion distortion loss, a second weight for semantic feature loss, and / or a third weight for semantic consistency loss. The first weight is used to describe the significance of motion distortion loss in the sample video frame sequence; the second weight is used to describe the significance of semantic feature loss in the sample video frame sequence; and the third weight is used to describe the significance of semantic consistency loss in the sample video frame sequence. The dynamic fusion module performs weighted processing on motion distortion loss, semantic feature loss, and / or semantic consistency loss according to the first weight, the second weight, and / or the third weight to obtain fusion loss features. Based on the fusion loss features, it outputs the quality prediction score corresponding to the training sample group.

[0222] Optionally, the video quality assessment model to be trained may also include a quality prediction network;

[0223] Training unit 502 is specifically used to input the fusion loss features into the quality prediction network. The quality prediction network performs dimensionality compression on the fusion loss features and outputs the quality prediction scores corresponding to the training sample groups.

[0224] The fifth embodiment of this application also provides a video quality assessment apparatus corresponding to the video quality assessment method embodiment provided in the second embodiment. Since the apparatus embodiment is basically similar to the method embodiment, it is described simply. For details of the relevant technical features and their effects, please refer to the corresponding descriptions of the video quality assessment method embodiments provided above.

[0225] like Figure 6 As shown, the training device 600 for the video quality assessment model provided in this embodiment includes: an acquisition unit 601 and a generation unit 602;

[0226] Acquisition unit 601 is used to acquire the target video frame sequence to be quality evaluated;

[0227] The generation unit 602 is used to input the target video frame sequence into the video quality assessment model, so that the video quality assessment model outputs the quality score corresponding to the target video frame sequence. The video quality assessment model is trained using the training method described in the first embodiment.

[0228] The sixth embodiment of this application also provides a video quality assessment apparatus corresponding to the video quality assessment method embodiment provided in the third embodiment. Since the apparatus embodiment is basically similar to the method embodiment, it is described simply. For details of the relevant technical features and their effects, please refer to the corresponding description of the video quality assessment method embodiment provided in the third embodiment above.

[0229] The training device 700 for the video quality assessment model provided in this embodiment includes: a reconstruction unit 701 and a determination unit 702;

[0230] The reconstruction unit 701 is used to perform content reconstruction processing on the target video frame sequence to be quality assessed, so as to obtain the reconstructed video frame sequence corresponding to the target video frame sequence.

[0231] The determining unit 702 is used to determine the semantic feature loss corresponding to the target video frame sequence based on the target video frame sequence and the reconstructed video frame sequence; the semantic feature loss is used to describe the degree of influence of the semantic discriminability of the sample video frame sequence on the quality of the sample video frame sequence.

[0232] The determination unit 702 is also used to determine the target quality score corresponding to the target video frame sequence based on the semantic feature loss.

[0233] The seventh embodiment of this application also provides an electronic device embodiment corresponding to the training method of the video quality assessment model provided in the first embodiment and the video quality assessment method provided in the second embodiment. The following description of the electronic device embodiment is merely illustrative. The electronic device embodiment is as follows:

[0234] Please refer to Figure 7 Understanding the above electronic devices, Figure 7 This is a schematic diagram of an electronic device. The electronic device 1000 provided in this embodiment includes: a processor 1001 and a memory 1002; the memory 1002 is used to store computer instructions for data processing, and when the computer instructions are read and executed by the processor 1001, they can execute any one of the steps in the method embodiments described in the first to third embodiments.

[0235] The eighth embodiment of this application also provides a computer-readable storage medium for implementing the methods of any one of the first to third embodiments. The computer-readable storage medium embodiments provided in this application are described in a relatively simple manner; relevant parts can be found in the corresponding descriptions of the above method embodiments. The embodiments described below are merely illustrative.

[0236] The computer-readable storage medium provided in this embodiment stores computer instructions, which, when executed by a processor, can perform any one of the steps in the methods provided in the first to third embodiments described above.

[0237] The ninth embodiment of this application also provides a computer program product for implementing the methods of the first to third embodiments. The computer program product embodiments provided in this application are described in a relatively simple manner; relevant parts can be found in the corresponding descriptions of the above method embodiments. The embodiments described below are merely illustrative. The computer program product provided in this embodiment includes a computer program, which, when executed by a processor, performs any one of the steps of the methods provided in the first to third embodiments described above.

[0238] The tenth embodiment of this application also provides a video quality assessment model trained using the training method of the video quality assessment model provided in the first embodiment above. The video quality assessment model embodiments provided in this application are described in a relatively simple manner; relevant parts can be found in the corresponding descriptions of the above method embodiments. The embodiments described below are merely illustrative. (Refer to...) Figure 2 The video quality assessment model 200 includes: motion distortion assessment module 201, semantic distortion assessment module 220, and / or semantic consistency assessment module 230;

[0239] Motion distortion evaluation module 210 is used to generate motion distortion loss for the target video frame sequence of the input video quality evaluation model; the motion distortion loss describes the degree of influence of the motion state of a preset object in the sample video frame sequence on the quality of the sample video frame sequence; semantic distortion evaluation module 220 is used to generate semantic feature loss for the target video frame sequence; the semantic feature loss describes the degree of influence of the semantic discriminability of the sample video frame sequence on the quality of the sample video frame sequence; semantic consistency evaluation module 230 is used to generate semantic consistency loss for the target video frame sequence; the semantic consistency loss describes the degree of influence of the semantic alignment between the sample video frame sequence corresponding to the training sample group and the quality description text corresponding to the sample video frame sequence on the quality of the sample video frame sequence.

[0240] Optionally, model 200 further includes: a dynamic fusion module 240; the dynamic fusion module 240 is used to generate a first weight for motion distortion loss, a second weight for semantic feature loss, and / or a first weight, a second weight, and / or a third weight for semantic consistency loss respectively; the first weight is used to describe the significance of motion distortion loss in the sample video frame sequence; the second weight is used to describe the significance of semantic feature loss in the sample video frame sequence; the third weight is used to describe the significance of semantic consistency loss in the sample video frame sequence; the dynamic fusion module 240 is also used to perform weighted processing on motion distortion loss, semantic feature loss, and / or semantic consistency loss according to the first weight, the second weight, and / or the third weight to obtain fused loss features.

[0241] Optionally, model 200 also includes: a quality prediction network 250; the quality prediction network 250 is used to perform dimensionality compression on the fusion loss features to output the quality prediction scores corresponding to the training sample groups.

[0242] The eleventh embodiment of this application also provides a video quality assessment model based on semantic distortion. The embodiments of the video quality assessment model based on semantic distortion provided in this application are described relatively simply; relevant parts can be found in the corresponding descriptions of the above method embodiments. The embodiments described below are merely illustrative. (Refer to...) Figure 2 The semantic distortion-based video quality assessment model is the semantic distortion assessment module 220 in the video quality assessment model 200. Specifically, the motion distortion assessment module 210, semantic distortion assessment module 220, and semantic consistency assessment module 230 in the video quality assessment model 200 can each independently form a video quality assessment model. Therefore, the semantic distortion-based video quality assessment model can at least include: a mask reconstruction unit 211 and a feature processing unit 222.

[0243] The mask reconstruction unit 221 is used to perform content reconstruction processing on the target video frame sequence to be evaluated for semantic distortion, so as to obtain the reconstructed video frame sequence corresponding to the target video frame sequence.

[0244] The feature processing unit 222 is used to determine the semantic feature loss corresponding to the target video frame sequence based on the target video frame sequence and the reconstructed video frame sequence; the semantic feature loss is used to describe the semantic distortion of the target video frame sequence.

[0245] In an optional implementation, the feature processing unit 222 may further include a first processing subunit, a second processing subunit and / or a third processing subunit, as well as a loss determination unit;

[0246] The system comprises: a first processing subunit for determining the semantic reconstruction bias between the target video frame sequence and the reconstructed video frame sequence; a second processing subunit for determining the classification stability of a preset object in the target video frame sequence in each reconstructed video frame; a third processing subunit for determining the video classification reconstruction bias between the sample video frame sequence and the reconstructed video frame sequence; and a loss determination unit for determining the semantic feature loss corresponding to the target video frame sequence based on the semantic reconstruction bias, classification stability, and / or video classification reconstruction bias. The semantic reconstruction bias and video classification reconstruction bias are positively correlated with the semantic feature loss, while the classification stability is negatively correlated with the semantic feature loss.

[0247] Optionally, the semantic distortion-based video quality assessment model may also include other functional modules, which can be referred to the previous description and will not be repeated here.

[0248] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0249] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0250] 1. Computer-readable media includes both permanent and non-permanent, removable and non-removable media, which can store information by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined in this application, computer-readable media does not include non-transitory computer-readable media, such as modulated data signals and carrier waves.

[0251] 2. Those skilled in the art will understand that embodiments of this application can provide methods, systems, or computer program products. Therefore, embodiments of this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, embodiments of this application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0252] 3. This application embodiment may involve the use of user data. In practical applications, user-specific personal data may be used within the scope permitted by applicable laws and regulations of the country in which the application is located (e.g., with the user's explicit consent and effective notification to the user, etc.). Furthermore, the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. The collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0253] Although this application discloses preferred embodiments as described above, it is not intended to limit this application. Any person skilled in the art can make possible changes and modifications without departing from the spirit and scope of this application. Therefore, the scope of protection of this application should be determined by the scope defined in the claims of this application.

Claims

1. A method for training a video quality assessment model, characterized in that, The video quality assessment model to be trained includes at least: a motion distortion assessment module, a semantic distortion assessment module, and a semantic consistency assessment module; the method includes: Obtain a training dataset; the training dataset includes at least one training sample group, each training sample group including at least: a sample video frame sequence and a set of descriptive text corresponding to the sample video frame sequence; the set of descriptive text includes at least one pre-defined video quality descriptive text for representing high-quality semantics; The video quality assessment model to be trained is trained using the training dataset according to the following training process: A training sample group from the training dataset is input into the video quality assessment model to be trained. The motion distortion assessment module generates a motion distortion loss corresponding to the training sample group, the semantic distortion assessment module generates a semantic feature loss, and the semantic consistency assessment module generates a semantic consistency loss. The motion distortion loss describes the degree of influence of the motion state of a preset object in the sample video frame sequence on the quality of the sample video frame sequence. The semantic feature loss describes the degree of influence of the semantic discriminability of the sample video frame sequence on the quality of the sample video frame sequence. The semantic consistency loss describes the degree of influence of the semantic alignment between the sample video frame sequence corresponding to the training sample group and the quality description text corresponding to the sample video frame sequence on the quality of the sample video frame sequence. The model parameters of the video quality assessment model to be trained are adjusted according to the motion distortion loss, semantic feature loss and semantic consistency loss corresponding to the preset number of training sample groups, respectively. The training process is repeated using the training sample group in the training dataset until the preset end-of-training condition is met, resulting in a trained video quality evaluation model.

2. The method of claim 1, wherein, The training sample set includes the real trajectory features of the preset object in the sample video frame sequence; the step of the motion distortion evaluation module generating the motion distortion loss corresponding to the training sample set includes: The sample video frame sequence from the training sample group is input into the motion distortion evaluation module, and the motion distortion evaluation module determines the predicted motion trajectory features of the preset object in the sample video frame sequence based on the sample video frame sequence. Determine the trajectory feature deviation between the predicted motion trajectory features of the preset object and the actual motion trajectory features of the preset object in the sample video frame sequence; The motion distortion loss of the preset object is determined based on the trajectory feature deviation; the trajectory feature deviation is used to describe the motion stability of the preset object, and the trajectory feature deviation is negatively correlated with the motion stability.

3. The method of claim 2, wherein, The method further includes: The motion distortion assessment module determines the visual features corresponding to the preset object in at least two different video frames adjacent or near each other in the sample video frame sequence based on the sample video frame sequence. Determine the first similarity between the visual features corresponding to the preset object in different video frames; the first similarity is negatively correlated with the degree of geometric deformation or motion blur of the preset object in the video frame sequence; The step of determining the motion distortion loss of the preset object based on the trajectory feature deviation includes: Based on the trajectory feature deviation and the first similarity, the motion distortion loss of the preset object is determined.

4. The method according to claim 2 or 3, characterized in that, The step of determining the motion distortion loss of the preset object based on the trajectory feature deviation and the first similarity includes: By fusing the trajectory feature deviation and the first similarity, the motion distortion loss of the preset object is obtained.

5. The method according to claim 2 or 3, characterized in that, The motion distortion assessment module includes a target tracking unit and a trajectory prediction unit; determining the predicted motion trajectory features of the preset object in the sample video frame sequence includes: S1: The target tracking unit generates a current state feature sequence corresponding to the preset object based on the historical state feature sequence of the preset object in the historical video frames before the current video frame and the current state feature in the current video frame; S2: The trajectory prediction unit determines the predicted state features of the preset object in the next video frame after the current video frame based on the current state feature sequence. The predicted state features include the predicted position features of the preset object in the next video frame. S3: When it is detected that the next video frame is not the last video frame in the sample video frame sequence, the next video frame is determined as the current video frame, and the execution of S1 to S3 is returned. S4: When the next video frame is detected to be the last video frame in the sample video frame sequence, the predicted motion trajectory features of the preset object are determined according to the predicted position features of the preset object in each video frame.

6. The method according to claim 5, characterized in that, Determining the trajectory feature deviation between the predicted motion trajectory features and the actual motion trajectory features of the preset object in the sample video frame sequence includes: For one of the predicted position features in the predicted motion trajectory features, calculate the position feature deviation between the predicted position feature and the actual position feature corresponding to the predicted position feature in the actual motion trajectory; The mean square error is calculated based on the position feature deviation corresponding to each predicted position feature in the predicted motion trajectory features, and the mean square error is determined as the trajectory feature deviation.

7. The method according to claim 1, characterized in that, The semantic distortion evaluation module includes a mask reconstruction unit and a feature processing unit; the step of generating semantic feature loss in the semantic distortion evaluation module includes: The sample video frame sequence in the training sample group is input into the mask reconstruction unit, and the sample video frame sequence is processed for content reconstruction to obtain the reconstructed video frame sequence corresponding to the sample video frame sequence. The feature processing unit determines the semantic feature loss corresponding to the training sample group based on the sample video frame sequence and the reconstructed video frame sequence.

8. The method according to claim 7, characterized in that, The mask reconstruction unit performs content reconstruction processing on the sample video frame sequence to obtain the reconstructed video frame sequence corresponding to the sample video frame sequence, including: The mask reconstruction unit performs block processing on each video frame in the sample video frame sequence to obtain the image block set corresponding to the sample video frame sequence. Based on a preset masking rate, a first type of image block and a second type of image block are determined in the image block set; the first type of image block is the image block to be masked, and the second type of image block is the image block that is not masked. The first type of image block is masked, and based on the second type of image block, the masked first type of image block is reconstructed in the corresponding video frame sequence to obtain the reconstructed video frame sequence corresponding to the sample video frame sequence.

9. The method according to claim 7, characterized in that, The sample video frame sequence includes at least one preset object; determining the semantic feature loss corresponding to the training sample group based on the sample video frame sequence and the reconstructed video frame sequence includes: Determine the semantic reconstruction deviation between the sample video frame sequence and the reconstructed video frame sequence; Determine the classification stability of the preset object in each reconstructed video frame of the reconstructed video frame sequence; and / or, determine the video classification reconstruction deviation between the sample video frame sequence and the reconstructed video frame sequence; The semantic feature loss corresponding to the training sample group is determined based on the semantic reconstruction bias, the classification stability, and / or the video classification reconstruction bias; wherein the semantic reconstruction bias and the video classification reconstruction bias are positively correlated with the semantic feature loss, and the classification stability is negatively correlated with the semantic feature loss.

10. The method according to claim 9, characterized in that, Determining the semantic reconstruction deviation between the sample video frame sequence and the reconstructed video frame sequence includes: Extract the first semantic features corresponding to the sample video frame sequence, and extract the second semantic features corresponding to the reconstructed video frame sequence; Determine the second similarity between the first semantic feature and the second semantic feature; The second similarity is determined as the semantic reconstruction deviation.

11. The method according to claim 9, characterized in that, Determining the classification stability of the preset object in each reconstructed video frame of the reconstructed video frame sequence includes: Determine the classification confidence level of the preset object in each reconstructed video frame of the reconstructed video frame sequence; Based on the classification confidence level of the preset object in each reconstructed video frame of the reconstructed video frame sequence, the confidence variance of the preset object in the reconstructed video frame sequence is determined; the confidence variance is used to describe the classification stability of the preset object in the reconstructed video frame sequence.

12. The method according to claim 11, characterized in that, Determining the classification confidence level of the preset object in each reconstructed video frame of the reconstructed video frame sequence includes: Each reconstructed video frame in the reconstructed video frame sequence is input into a first classification model, and the first classification model generates a classification probability for each reconstructed video frame; the classification probability is used to represent the classification confidence of the first classification model for the classification result of the preset object in each reconstructed video frame.

13. The method according to claim 9, characterized in that, The determination of the video classification reconstruction bias between the sample video frame sequence and the reconstructed video frame sequence includes: The sample video frame sequence and the reconstructed video frame sequence are input into a second classification model. The second classification model generates a first classification probability distribution and a second classification probability distribution for the sample video frame sequence and the reconstructed video frame sequence, respectively. The first classification probability distribution includes the probability value of at least one video category to which the sample video frame sequence belongs. The second classification probability distribution includes the probability value of at least one video category to which the reconstructed video frame sequence belongs. The distribution difference between the first classification probability distribution and the second classification probability distribution is determined, and the distribution difference is determined as the video classification reconstruction bias.

14. The method according to claim 9, characterized in that, The step of determining the semantic feature loss corresponding to the training sample group based on the semantic reconstruction bias, the classification stability, or the video classification reconstruction bias includes: The semantic reconstruction bias is determined as the semantic feature loss corresponding to the training sample group; The classification stability is determined as the semantic feature loss corresponding to the training sample group; or... The video classification and reconstruction bias is determined as the semantic feature loss corresponding to the training sample group.

15. The method according to claim 9, characterized in that, The step of determining the semantic feature loss corresponding to the training sample group based on the semantic reconstruction bias, the classification stability, and the video classification reconstruction bias includes: The semantic reconstruction bias, the classification stability, and the video classification reconstruction bias are combined to obtain the fused features; The fused features are determined as the semantic feature loss corresponding to the training sample group.

16. The method according to claim 1, characterized in that, The step of generating the semantic consistency loss corresponding to the training sample group by the semantic consistency evaluation module includes: The training sample group is input into the semantic consistency evaluation module, which determines the visual semantic features of the sample video frame sequence and the textual semantic features of the quality description text in the description text set. Based on the visual semantic features and the text semantic features, determine the semantic consistency loss corresponding to the training sample group.

17. The method according to claim 16, characterized in that, The step of determining the semantic consistency loss corresponding to the training sample group based on the visual semantic features and the text semantic features includes: Calculate the third similarity between the visual semantic features and the text semantic features; The third similarity is determined as the semantic consistency loss corresponding to the training sample group; the third similarity is negatively correlated with the semantic consistency loss.

18. The method according to claim 1, characterized in that, The training sample set also includes the target quality score corresponding to the sample video frame sequence; adjusting the model parameters of the video quality assessment model to be trained according to the motion distortion loss, semantic feature loss, and / or semantic consistency loss corresponding to the preset number of training sample sets includes: Based on the motion distortion loss, semantic feature loss, and / or semantic consistency loss corresponding to the preset number of training sample groups, output the quality prediction score corresponding to each training sample group; The model parameters of the video quality assessment model to be trained are adjusted based on the principle of reducing the difference between the predicted quality score and the target quality score corresponding to each training sample group.

19. The method according to claim 18, characterized in that, The video quality assessment model to be trained further includes a dynamic fusion module; the step of outputting the quality prediction score corresponding to the training sample group based on the motion distortion loss, the semantic feature loss, and / or the semantic consistency loss corresponding to the training sample group includes: The motion distortion loss, the semantic feature loss, and / or the semantic consistency loss are input into the dynamic fusion module. The dynamic fusion module generates a first weight for the motion distortion loss, a second weight for the semantic feature loss, and / or a third weight for the semantic consistency loss. The first weight describes the significance of the motion distortion loss in the sample video frame sequence; the second weight describes the significance of the semantic feature loss in the sample video frame sequence; and the third weight describes the significance of the semantic consistency loss in the sample video frame sequence. The dynamic fusion module performs weighted processing on the motion distortion loss, the semantic feature loss, and / or the semantic consistency loss according to the first weight, the second weight, and / or the third weight to obtain the fusion loss features; Output the quality prediction score corresponding to the training sample group based on the fusion loss features.

20. The method according to claim 19, characterized in that, The video quality assessment model to be trained further includes a quality prediction network; the step of outputting the quality prediction score corresponding to the training sample group based on the fusion loss features includes: The fusion loss features are input into the quality prediction network, which performs dimensionality compression on the fusion loss features and outputs the quality prediction scores corresponding to the training sample groups.

21. A video quality assessment method, characterized in that, The method includes: Obtain the target video frame sequence to be evaluated for quality. The target video frame sequence is input into a video quality assessment model so that the video quality assessment model outputs a quality score corresponding to the target video frame sequence; wherein the video quality assessment model is trained according to the training method of any one of claims 1 to 20.

22. A video quality assessment method, characterized in that, The method is performed using a video quality assessment model trained according to the training method of any one of claims 1 to 20, the method comprising: Content reconstruction processing is performed on the target video frame sequence to be quality assessed to obtain the reconstructed video frame sequence corresponding to the target video frame sequence. Based on the target video frame sequence and the reconstructed video frame sequence, the semantic feature loss corresponding to the target video frame sequence is determined; the semantic feature loss is used to describe the degree to which the semantic discriminability of the target video frame sequence affects the quality of the target video frame sequence. The target quality score corresponding to the target video frame sequence is determined based on the semantic feature loss.

23. The method according to claim 22, characterized in that, The step of determining the semantic feature loss corresponding to the target video frame sequence based on the target video frame sequence and the reconstructed video frame sequence includes: Determine the semantic reconstruction deviation between the target video frame sequence and the reconstructed video frame sequence; Determine the classification stability of a preset object in the target video frame sequence in each reconstructed video frame of the reconstructed video frame sequence; and / or, determine the video classification reconstruction deviation between the target video frame sequence and the reconstructed video frame sequence; The semantic feature loss corresponding to the target video frame sequence is determined based on the semantic reconstruction bias, the classification stability, and / or the video classification reconstruction bias; wherein the semantic reconstruction bias and the video classification reconstruction bias are positively correlated with the semantic feature loss, and the classification stability is negatively correlated with the semantic feature loss.

24. The method according to claim 22, characterized in that, The method further includes: Determine the motion distortion loss corresponding to the target video frame sequence; the motion distortion loss is used to describe the degree of influence of the motion state of a preset object in the target video frame sequence on the quality of the target video frame sequence; Determining the target quality score corresponding to the target video frame sequence based on the semantic feature loss includes: The target quality score corresponding to the target video frame sequence is determined based on the motion distortion loss and / or the semantic feature loss.

25. The method according to claim 24, characterized in that, The method further includes: Obtain the quality description text corresponding to the target video frame sequence; Based on the quality description text and the target video frame sequence, determine the semantic consistency loss corresponding to the target video frame sequence; the semantic consistency loss is used to describe the degree of influence of the semantic alignment between the target video frame sequence and the quality description text corresponding to the target video frame sequence on the quality of the target video frame sequence; Determining the target quality score corresponding to the target video frame sequence based on the motion distortion loss and / or the semantic feature loss includes: The target quality score corresponding to the target video frame sequence is determined based on the motion distortion loss, the semantic feature loss, and / or the semantic consistency loss.

26. The method according to claim 25, characterized in that, The step of determining the target quality score corresponding to the target video frame sequence based on the motion distortion loss, the semantic feature loss, and / or the semantic consistency loss includes: Determine the first weight corresponding to the motion distortion loss, the second weight corresponding to the semantic feature loss, and / or the third weight corresponding to the semantic consistency loss; Based on the first weight, the second weight, and / or the third weight, the motion distortion loss, the semantic feature loss, and / or the semantic consistency loss are weighted to obtain the fusion loss feature; The target quality score corresponding to the target video frame sequence is determined based on the fusion loss features.

27. A training device for a video quality assessment model, characterized in that, The video quality assessment model to be trained includes at least: a motion distortion assessment module, a semantic distortion assessment module, and a semantic consistency assessment module; the device includes an acquisition unit and a training unit. The acquisition unit is used to acquire a training dataset; the training dataset includes at least one training sample group, and each training sample group includes at least: a sample video frame sequence and a set of descriptive text corresponding to the sample video frame sequence; the set of descriptive text includes at least one pre-set video quality descriptive text for representing high-quality semantics; The training unit is configured to train the video quality assessment model to be trained based on the training dataset using the following training process: A training sample group from the training dataset is input into the video quality assessment model to be trained; the motion distortion assessment module generates a motion distortion loss corresponding to the training sample group; the semantic distortion assessment module generates a semantic feature loss; and the semantic consistency assessment module generates a semantic consistency loss. The motion distortion loss describes the degree of influence of the motion state of a preset object in the sample video frame sequence on the quality of the sample video frame sequence. The semantic feature loss describes the degree of influence of the semantic discriminability of the sample video frame sequence on the quality of the sample video frame sequence. The semantic consistency loss describes the degree of influence of the semantic alignment between the sample video frame sequence corresponding to the training sample group and the quality description text corresponding to the sample video frame sequence on the quality of the sample video frame sequence. The model parameters of the video quality assessment model to be trained are adjusted according to the motion distortion loss, semantic feature loss, and semantic consistency loss corresponding to a preset number of training sample groups. The training process is repeated using a sufficient number of training sample groups from the training dataset until a preset end-of-training condition is met, resulting in the trained video quality assessment model.

28. A video quality assessment device, characterized in that, The device includes an acquisition unit and a generation unit; The acquisition unit is used to acquire the target video frame sequence to be evaluated for quality. The generation unit is used to input the target video frame sequence into the video quality assessment model so that the video quality assessment model outputs the quality score corresponding to the target video frame sequence; the video quality assessment model is trained according to the training method of any one of claims 1 to 20.

29. A video quality assessment device, characterized in that, The video quality assessment device is used to run a video quality assessment model trained according to the training method of any one of claims 1 to 20, and the device includes a reconstruction unit and a determination unit. The reconstruction unit is used to perform content reconstruction processing on the target video frame sequence to be quality evaluated, so as to obtain the reconstructed video frame sequence corresponding to the target video frame sequence. The determining unit is configured to determine the semantic feature loss corresponding to the target video frame sequence based on the target video frame sequence and the reconstructed video frame sequence; the semantic feature loss is used to describe the degree to which the semantic discriminability of the target video frame sequence affects the quality of the target video frame sequence. The determining unit is further configured to determine the target quality score corresponding to the target video frame sequence based on the semantic feature loss.

30. An electronic device, characterized in that, include: Processor, memory, and computer program instructions stored in said memory and executable on the processor; When the processor executes the computer program instructions, it implements the method as described in any one of claims 1-26.

31. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1-26.

32. A computer program product comprising a computer program that, when executed by a processor, implements the method as described in any one of claims 1-26.

33. A video quality assessment model, characterized in that, The video quality assessment model is trained according to the training method described in any one of claims 1 to 20, and the video quality assessment model includes: a motion distortion assessment module, a semantic distortion assessment module, and a semantic consistency assessment module; The motion distortion assessment module is used to generate motion distortion loss for the target video frame sequence input to the video quality assessment model; the motion distortion loss is used to describe the degree of influence of the motion state of a preset object in the target video frame sequence on the quality of the target video frame sequence; The semantic distortion evaluation module is used to generate semantic feature loss for the target video frame sequence; the semantic feature loss is used to describe the degree to which the semantic discriminability of the target video frame sequence affects the quality of the target video frame sequence. The semantic consistency evaluation module is used to generate a semantic consistency loss for the target video frame sequence; the semantic consistency loss is used to describe the degree of influence of the semantic alignment between the target video frame sequence and the quality description text corresponding to the target video frame sequence on the quality of the target video frame sequence.

34. The model according to claim 33, characterized in that, The model also includes: a dynamic fusion module; The dynamic fusion module is used to generate a first weight for the motion distortion loss, a second weight for the semantic feature loss, and / or a third weight for the semantic consistency loss; the first weight is used to describe the significance of the motion distortion loss in the target video frame sequence; the second weight is used to describe the significance of the semantic feature loss in the target video frame sequence; and the third weight is used to describe the significance of the semantic consistency loss in the target video frame sequence. The dynamic fusion module is further configured to perform weighted processing on the motion distortion loss, the semantic feature loss, and / or the semantic consistency loss according to the first weight, the second weight, and / or the third weight, to obtain fusion loss features.

35. The model according to claim 34, characterized in that, The model also includes: a quality prediction network; The quality prediction network is used to perform dimensionality compression processing on the fusion loss features to output the quality prediction score corresponding to the target video frame sequence.

36. The model according to claim 33, characterized in that, The semantic distortion assessment module includes: a mask reconstruction unit and a feature processing unit; The mask reconstruction unit is used to perform content reconstruction processing on the target video frame sequence to be evaluated for semantic distortion, so as to obtain the reconstructed video frame sequence corresponding to the target video frame sequence. The feature processing unit is used to determine the semantic feature loss corresponding to the target video frame sequence based on the target video frame sequence and the reconstructed video frame sequence; the semantic feature loss is used to describe the semantic distortion of the target video frame sequence.

37. The model according to claim 36, characterized in that, The feature processing unit includes a first processing subunit, a second processing subunit and / or a third processing subunit, and a loss determination unit; The first processing subunit is configured to determine the semantic reconstruction deviation between the target video frame sequence and the reconstructed video frame sequence; The second processing subunit is used to determine the classification stability of a preset object in the target video frame sequence in each reconstructed video frame of the reconstructed video frame sequence; The third processing subunit is used to determine the video classification and reconstruction deviation between the target video frame sequence and the reconstructed video frame sequence; The loss determination unit is used to determine the semantic feature loss corresponding to the target video frame sequence based on the semantic reconstruction deviation, the classification stability, and / or the video classification reconstruction deviation; wherein the semantic reconstruction deviation and the video classification reconstruction deviation are positively correlated with the semantic feature loss, and the classification stability is negatively correlated with the semantic feature loss.

Images