Video quality detection method and system

Abstract

The embodiment of the invention provides a video quality detection method and system. In the method, after a video quality detection system obtains a target video, a single image frame is taken as granularity, and feature extraction is carried out on at least part of first image frames in the target video to obtain spatial domain features; and performing feature extraction on the second at least partial image frame in the target video by taking the image frame sequence as granularity to obtain a time domain feature and a spatial-temporal feature. And further, the video quality detection system generates fusion features based on the spatial domain features, the time domain features and the spatial-temporal features, and determines a quality detection result of the target video based on the fusion features.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a video quality detection method and system. Background Art

[0002] With the rapid development of the Internet and the popularization of video content, video has become an indispensable part of people's lives. In order to evaluate whether the video content meets the viewing needs of the audience, it is necessary to evaluate the video quality. Video quality evaluation refers to the process of objectively evaluating the image quality and overall look and feel in the video content, so as to determine whether the visual experience of the video meets the user's expectations. At the same time, it can help producers and video platforms understand the audience's satisfaction with the video content, thereby improving video production and enhancing user experience. The existing technology generally evaluates video quality through subjective evaluation methods, such as obtaining their evaluation of video quality by asking subjects to watch videos and fill out questionnaires. However, this subjective evaluation method is inefficient, has high labor costs, and the accuracy of the evaluation results cannot be guaranteed.

[0003] Therefore, it is necessary to provide a method that can automatically and accurately detect video quality.

[0004] The content of the background technology section is only the information known to the inventor personally, and does not mean that the above information has entered the public domain before the filing date of this specification, nor does it mean that it can become the prior art of this specification. Summary of the invention

[0005] This specification provides a video quality detection method and system, which can be applied to video quality evaluation scenarios.

[0006] In a first aspect, the present specification provides a video quality detection method, comprising: obtaining a target video, wherein the target video includes multiple image frames; taking a single image frame as the granularity, performing feature extraction on a first at least portion of the image frames in the target video to obtain spatial features, wherein the spatial features characterize static information within a single image frame in the target video; taking an image frame sequence as the granularity, performing feature extraction on a second at least portion of the image frames in the target video to obtain temporal features and spatiotemporal features, wherein the temporal features characterize dynamic change information between image frames in the target video, and the spatiotemporal features simultaneously characterize static information within a single image frame in the target video and dynamic change information between image frames; and generating fusion features based on the spatial features, the temporal features and the spatiotemporal features, and determining a quality detection result of the target video based on the fusion features.

[0007] In a second aspect, the present specification also provides a video quality detection system, comprising at least one storage medium and at least one processor, wherein the at least one storage medium stores at least one instruction set for performing a video quality detection method; the at least one processor is communicatively connected to the at least one storage medium, wherein the at least one processor reads the at least one instruction set when running, and executes any method as described in the first aspect according to the instructions of the at least one instruction set.

[0008] In a third aspect, the present specification also provides a computer-readable non-volatile storage medium, wherein the computer-readable non-volatile storage medium stores at least one instruction set, and when the at least one instruction set is executed by at least one processor, the video quality detection method as described in any one of the first aspects above is implemented.

[0009] It can be seen from the above technical solutions that the video quality detection method and system provided in this specification can extract the spatial domain features that can characterize the static information inside the single image frame in the target video based on the first at least part of the image frame, with a single image frame as the granularity, after acquiring the target video. With the image frame sequence as the granularity, based on the second at least part of the image frame, extract the temporal domain features that can characterize the dynamic change information between each image frame in the target video, and the spatiotemporal features that can simultaneously characterize the static information inside the single image frame in the target video and the dynamic change information between each image frame. Fusion features are generated based on spatial features, temporal features and spatiotemporal features, and the quality detection result of the target video is determined based on the fusion features. Such a video quality detection method not only realizes the automatic detection of the quality of the target video, but also the quality detection result of the target video is obtained based on the fusion features after the spatial features, temporal features and spatiotemporal features are fused, so that the determination of the quality detection result fully considers the features of different dimensions, and improves the accuracy and reliability of the quality detection result.

[0010] Other functions of the video quality detection method and system provided in this specification will be partially listed in the following description. The creative aspects of the video quality detection method and system provided in this specification can be fully explained by practicing or using the methods, devices and combinations described in the following detailed examples. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the technical solutions in the embodiments of this specification, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0012] Figure 1A schematic diagram of an application scenario of a video quality detection system provided according to an embodiment of this specification is shown;

[0013] Figure 2 A schematic diagram of the hardware structure of a computing device provided according to some embodiments of this specification is shown;

[0014] Figure 3 A schematic diagram of a process of a video quality detection method provided according to an embodiment of this specification is shown;

[0015] Figure 4 A schematic flow chart of a video quality detection method provided according to another embodiment of the present specification is shown;

[0016] Figure 5 A schematic diagram of a flow chart of a video quality detection method provided according to another embodiment of the present specification is shown;

[0017] Figure 6 A schematic diagram showing the structure of a target video provided according to an embodiment of the present specification is shown; and

[0018] Figure 7 A structural diagram of a video quality detection model is shown. DETAILED DESCRIPTION

[0019] The following description provides specific application scenarios and requirements of this specification, with the purpose of enabling those skilled in the art to make and use the contents of this specification. Various local modifications to the disclosed embodiments will be apparent to those skilled in the art, and the general principles defined herein may be applied to other embodiments and applications without departing from the spirit and scope of this specification. Therefore, this specification is not limited to the embodiments shown, but to the widest scope consistent with the claims.

[0020] The terms used herein are only used for the purpose of describing specific example embodiments and are not restrictive. For example, unless the context clearly indicates otherwise, as used herein, the singular forms "a", "an" and "the" may also include plural forms. When used in this specification, the terms "include", "comprise" and / or "contain" mean that the associated integers, steps, operations, elements and / or components exist, but do not exclude the existence of one or more other features, integers, steps, operations, elements, components and / or groups or that other features, integers, steps, operations, elements, components and / or groups may be added in the system / method.

[0021] In view of the following description, these and other features of the present specification, as well as the operation and function of the related elements of the structure, and the economy of the combination and manufacture of the parts can be significantly improved. Reference is made to the accompanying drawings, all of which form a part of this specification. However, it should be clearly understood that the drawings are for illustration and description purposes only and are not intended to limit the scope of this specification. It should also be understood that the drawings are not drawn to scale.

[0022] The flowcharts used in this specification illustrate the operations implemented by the system according to some embodiments in this specification. It should be clearly understood that the operations of the flowcharts may not be implemented in sequence. On the contrary, the operations may be implemented in reverse order or simultaneously. In addition, one or more other operations may be added to the flowchart. One or more operations may be removed from the flowchart.

[0023] In this specification, "X includes at least one of A, B or C" means that X includes at least A, or X includes at least B, or X includes at least C. That is, X may include only any one of A, B, and C, or may include any combination of A, B, and C and other possible contents / elements at the same time. The any combination of A, B, and C may be A, B, C, AB, AC, BC, or ABC.

[0024] In this specification, unless explicitly stated otherwise, the association relationship between structures can be a direct association relationship or an indirect association relationship. For example, when describing "A is connected to B", unless it is explicitly stated that A is directly connected to B, it should be understood that A can be directly connected to B or indirectly connected to B; for another example, when describing "A is above B", unless it is explicitly stated that A is directly above B (AB is adjacent and A is above B), it should be understood that A can be directly above B or indirectly above B (AB is separated by other elements and A is above B). And so on.

[0025] It should be noted that the user data obtained in this manual is authorized by the user and does not involve user privacy.

[0026] For the convenience of description, the terms that will appear in the following text of this specification are first explained.

[0027] Video Quality Assessment (VQA): refers to the process of objectively or subjectively evaluating the quality of video content. VQA can be used to judge the clarity, smoothness, color reproduction and other aspects of a video.

[0028] Image Quality Assessment (IQA): refers to the process of objectively or subjectively evaluating the quality of an image. IQA is mainly used to evaluate the possible distortion of an image during transmission, storage, and processing.

[0029] In the video quality evaluation scenario, it is generally necessary to evaluate the video quality, for example, by objectively evaluating the image quality in the video content and the overall look and feel of the video content, to determine whether the visual experience of the video meets the user's expectations, and at the same time help the production staff and the video platform understand the user's satisfaction with the video content, thereby improving video production and enhancing user experience. In the prior art, the video quality is generally evaluated by subjective evaluation, for example, by asking the evaluator to watch the video and fill in the evaluation form to obtain the evaluator's evaluation of the video quality. Among them, the evaluator can be an operation and maintenance personnel, or a user who watches the video. However, this subjective evaluation method consumes a lot of manpower and time costs, and different evaluators may come to different evaluation conclusions for the same video, that is, the method of determining the evaluation results of the video screen through subjective evaluation cannot guarantee the consistency of the evaluation criteria in the process of evaluating the video, and the accuracy of the video evaluation results is affected by human subjective factors. As a result, the efficiency of video quality evaluation in the prior art is low, and the accuracy of video quality evaluation cannot be guaranteed.

[0030] The video quality detection method provided in this specification can determine the spatial domain features used to characterize the static information inside a single image frame in the target video with a single image frame as the granularity after acquiring the target video including multiple image frames. With the image frame sequence as the granularity, determine the temporal features used to characterize the dynamic change information between each image frame in the target video, and the spatiotemporal features used to characterize the static information inside a single image frame in the target video and the dynamic change information between each image frame. Extract different features in the target video from different granularities, fuse the extracted spatial domain features, temporal domain features and spatiotemporal features to obtain fused features, and determine the quality detection result of the target video based on the fused features. In this way of determining the quality detection result, since the fused features are obtained by fusion of features in multiple dimensions based on spatial domain features, temporal domain features and spatiotemporal features, the fused features can more comprehensively reflect the characteristics of the target video. When the quality detection result is determined based on the fused features in the subsequent step, the quality detection result obtained is also more accurate. In addition, the evaluation process is not affected by subjective factors, and the quality of the video can be determined more accurately.

[0031] It should be noted that the above description of the application scenario is only one of the multiple usage scenarios provided in this specification. Those skilled in the art should understand that when the video quality detection method and system provided in this specification are applied to other usage scenarios, their implementation methods and technical effects are similar.

[0032] Figure 1 A schematic diagram of an application scenario 100 of a video quality detection system 130 provided according to an embodiment of this specification is shown.

[0033] like Figure 1 As shown, the application scenario 100 may include a system 130 (hereinafter referred to as system 130). After acquiring the target video, the system 130 extracts the spatial features of the target video with a single image frame as the granularity. The temporal features and spatiotemporal features of the target video are extracted with an image frame sequence as the granularity. The fusion features of the target video are obtained based on the extracted spatial features, temporal features and spatiotemporal features, and the quality detection result of the target video is determined based on the fusion features. Through the above-mentioned method, the features of multiple aspects of the target video are extracted from different granularities (single image frame granularity, image frame sequence granularity), and the fusion features are obtained based on the extracted features of multiple aspects, so as to determine the quality detection result of the target video based on the fusion features containing multiple dimensional features. Thereby, the quality detection of the target video is automatically and accurately performed, the efficiency of quality detection is greatly improved, and the accuracy of quality detection result determination is improved.

[0034] Compared with the prior art method of evaluating video quality through subjective evaluation, the method of determining the quality test result of the target video proposed in this specification can determine the quality test result of the target video through objective evaluation, so as to ensure that the test standards for video quality test of different videos are the same, thus avoiding the subjective arbitrariness of manual evaluation. In other words, by determining the quality test result of the target video according to the video quality test method provided in this specification, the influence of human factors can be reduced, making the quality test result more objective and consistent.

[0035] In addition, compared with the subjective evaluation method of video quality, the quality detection result of the target video in the video quality detection method provided in this specification is automatically determined, which not only ensures the consistency of the evaluation criteria when evaluating different videos, but also the process of evaluating the video does not require manual viewing and evaluation, which greatly improves the efficiency of video quality evaluation and reduces the consumption of manpower and time costs. In addition, the method provided in this specification can be applied to different types of video quality evaluation scenarios, and its scalability is strong.

[0036] The system 130 may be an electronic device with certain computing capabilities. The system 130 may execute the video quality detection method provided in this specification. The system 130 may store data or instructions for executing the video quality detection method described in this specification, and may execute or be used to execute the data or instructions. The system 130 may include a hardware device with data information processing functions and the necessary programs required to drive the hardware device to work.

[0037] The system 130 may be a single computing device or a cluster system composed of multiple computing devices. The data or instructions stored in the system 130 for executing the video quality detection method described in this specification may adopt any form of system architecture, such as a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture.

[0038] The above system architecture may include a terminal and a server. The terminal may be deployed with a client of the system, and the server may be deployed with a server of the system. The terminal may include a mobile device, a tablet computer, a laptop computer or the like, or any combination thereof. In some embodiments, the mobile device may include a smart home device, a smart mobile device, a virtual reality device, an augmented reality device or the like, or any combination thereof. In some embodiments, the smart home device may include a smart TV, a desktop computer, etc., or any combination thereof. In some embodiments, the smart mobile device may include a smart phone, a personal digital assistant, a gaming device, a navigation device, etc., or any combination thereof.

[0039] In some embodiments, the system 130 can be used as a terminal in the above system architecture. For example, the system 130 can obtain a target video, and obtain a quality detection result of the target video by executing the video quality detection method provided in this specification, so as to realize automatic and accurate detection of the quality of the target video.

[0040] In some embodiments, the video quality detection system 130 can be used as a server in the above system architecture. For example, the terminal sends the target video to the video quality detection system 130, and the video quality detection system 130 executes the video quality detection method provided in this specification, extracts multiple features of the target video, generates fusion features based on the features of multiple aspects, and determines the quality detection result of the target video based on the fusion features. Then, the video quality detection system 130 can feed back the quality detection result of the target video to the terminal.

[0041] Figure 2 A hardware structure diagram of a computing device 200 provided according to some embodiments of this specification is shown. The computing device 200 can be used as Figure 1In some embodiments, when the video quality detection system 130 uses a device cluster, the computing device 200 can be used as any device in the video quality detection system 130.

[0042] like Figure 2 As shown, computing device 200 includes at least one storage medium 230 and at least one processor 220. In some embodiments, computing device 200 may further include an internal communication bus 210. In some embodiments, computing device 200 may further include a communication port 250. In some embodiments, computing device 200 may further include an I / O component 260.

[0043] Internal communication bus 210 may connect various system components, including storage media 230 and processor 220. I / O components 260 support input / output between computing device 200 and other components.

[0044] The communication port 250 is used for data communication between the computing device 200 and the outside world. For example, the computing device 200 can be connected to a network through the communication port 250.

[0045] The storage medium 230 may include a data storage device. The data storage device may be a non-temporary storage medium or a temporary storage medium. For example, the data storage device may include one or more of a disk 232, a read-only storage medium (ROM) 234, or a random access storage medium (RAM) 236. The storage medium 230 also includes at least one instruction set stored in the data storage device. The instruction set is a computer program code, and the computer program code may include a program, a routine, an object, a component, a data structure, a process, a module, etc. for executing the video quality detection method provided in this specification.

[0046] At least one processor 220 is connected to at least one storage medium 230 through an internal communication bus 210. At least one processor 220 is used to execute the at least one instruction set. When the system 130 is running, at least one processor 220 reads at least one instruction set and executes the video quality detection method provided in this specification according to the instructions of at least one instruction set.

[0047] The processor 220 can execute all the steps included in the video quality detection method. The processor 220 can be in the form of one or more processors. The processor 220 can issue execution instructions. The processor 220 may include one or more hardware processors, such as a microcontroller, a microprocessor, a reduced instruction set computer (RISC), an application-specific integrated circuit (ASIC), an application-specific instruction set processor (ASIP), a central processing unit (CPU), a graphics processing unit (GPU), a physical processing unit (PPU), a microcontroller unit, a digital signal processor (DSP), a field programmable gate array (FPGA), an advanced RISC machine (ARM), a programmable logic device (PLD), any circuit or processor capable of performing one or more functions, etc., or any combination thereof.

[0048] For illustrative purposes only, only one processor 220 is shown in the computing device 200 in the drawings of this specification. However, it should be noted that the computing device 200 in this specification may also include multiple processors, and therefore, the operations and / or method steps disclosed in this specification may be performed by one processor as in this specification, or may be performed jointly by multiple processors. For example, if the processor 220 of the computing device 200 in this specification performs step A and step B, it should be understood that step A and step B may also be performed jointly or separately by two different processors 220 (e.g., the first processor performs step A, the second processor performs step B, or the first and second processors perform steps A and B together).

[0049] Figure 3 A schematic diagram of a process of a video quality detection method provided according to an embodiment of this specification is shown; Figure 4 FIG. 1 is a flow chart of a video quality detection method provided according to another embodiment of the present specification; the video quality detection method P300 may be executed by the video quality detection system 130. Figure 3 Shown and Figure 4 As shown, the method P300 provided in this specification may include S310-S370, wherein:

[0050] S310: Obtain a target video, where the target video includes multiple image frames.

[0051] In the embodiments of this specification, the application scenario is video quality detection under a video application as an example for explanation. The method for obtaining the target video can be to obtain the target video uploaded by the user, and then perform video quality detection on the target video. When the quality detection result of the target video meets the preset requirements, it is determined that the target video passes the quality detection, and the user can upload the target video to the video application. Alternatively, it can also be to obtain the video that has been uploaded in the video application and has not been subjected to video quality detection. When the quality detection result of the target video meets the preset requirements, the target video is retained on the video application, otherwise the target video is removed from the shelf. It should be understood that the above embodiments are only exemplary descriptions, and the specific application scenarios can be flexibly adjusted according to user needs, and are not limited to the above embodiments. The video quality detection method provided in this specification can be applied to any scene where video quality needs to be detected. The method for obtaining the target video can also be to obtain the video shot by the user, or to obtain the video selected and uploaded by the user from the album, or to obtain the downloaded video, etc.

[0052] S330: Taking a single image frame as a granularity, extract features from a first at least part of the image frames in the target video to obtain spatial features, where the spatial features represent static information inside a single image frame in the target video.

[0053] In an embodiment of the present specification, the spatial dimension may include multiple sub-spatial dimensions. When determining the spatial feature, feature extraction may be performed on the first at least part of the image frame in multiple sub-spatial dimensions to obtain sub-spatial features corresponding to each of the multiple sub-spatial dimensions, and then the sub-spatial features corresponding to the multiple sub-spatial dimensions are spliced ​​to obtain the spatial feature. Among them, the multiple sub-spatial dimensions include at least two of the following: semantic dimension, distortion dimension, edge dimension, or comprehensive dimension. In an embodiment of the present specification, the extraction of spatial features may be based on a key frame sequence extracted by an IQA model, and the IQA model uses multiple pre-trained models to extract multiple sub-spatial dimensions from the key frame sequence to obtain the spatial feature based on the extracted multiple sub-spatial features.

[0054] The method of splicing the sub-spatial features corresponding to multiple sub-spatial dimensions is, for example, to directly splice each sub-spatial feature to obtain the spatial feature. Alternatively, the target sub-spatial features corresponding to each sub-spatial dimension may be spliced ​​according to the weight information corresponding to each sub-spatial dimension (different sub-spatial dimensions may correspond to different weight information) to obtain the spatial feature. Among them, since different spatial features may correspond to different dimensions, before splicing the sub-spatial features, it is necessary to unify the dimensions of each sub-spatial feature. After the dimensions of the sub-spatial features are unified, the sub-spatial features of the unified dimension are spliced ​​to obtain the spatial feature.

[0055] Still taking the weight information corresponding to each sub-spatial dimension as an example for explanation, the weight information can be pre-trained by iteration, for example. Determined in advance based on a large number of video sample-quality score sets, different sub-space dimensions correspond to a pre-trained weight information, which is used to indicate the importance of each sub-space dimension for subsequent video quality detection. By splicing the sub-spatial features according to the weight information to obtain the spatial features, it can be ensured that the more important sub-spatial features eventually account for a larger proportion of the spatial features.

[0056] Among them, for any first sub-spatial domain dimension among the multiple sub-spatial domain dimensions, the method for determining the corresponding sub-spatial domain feature can be: in the first sub-spatial domain dimension, feature extraction is performed on each image frame in the first at least partial image frame respectively to obtain the frame feature of each image frame in the first sub-spatial domain dimension; and the frame features of the first at least partial image frame in the first sub-spatial domain dimension are averaged to obtain the sub-spatial domain feature corresponding to the first sub-spatial domain dimension. It should be understood that the above embodiment is only an exemplary description, and the method for determining the sub-spatial domain feature can also be to obtain the sub-spatial domain feature corresponding to the first sub-spatial domain dimension by taking the median or performing clustering based on the frame features of the first at least partial image frame. The specific method for determining the sub-spatial domain feature can be flexibly adjusted according to user needs and is not limited to the above embodiment.

[0057] When extracting spatial features from a target video, processing the entire target video may be very time-consuming, especially when the target video is a long video, the speed of feature extraction is very slow. In order to increase the speed of extracting spatial features, in the embodiments of this specification, feature extraction is performed only based on the first at least a portion of the image frames in the target video to increase the speed of feature extraction.

[0058] In some possible embodiments, a method for determining the first at least partial image frame in a target video may be: after acquiring the target video, the target video may be divided into multiple video segments, and at least one key frame may be extracted from each of the multiple video segments to obtain the first at least partial image frame.

[0059] The target video may be divided into a plurality of video segments by taking a preset frame rate r as a division standard and dividing the target video into T video segments according to the total number of frames n of the target video, where T = n / r, wherein n and r are both positive integers, and n is greater than r. A key frame is extracted from each of the T segments, and a first at least partial image frame is formed based on the extracted T key frames. The frame set formed by the first at least partial image frame may also be referred to as a key frame sequence {I0, I1, …, I T-1}.

[0060] In some possible embodiments, the first image frame in each video clip can be selected as the key frame corresponding to each video clip. Alternatively, an image frame at a fixed position in each video clip (for example, the second image frame, the fifth image frame, etc.) can be selected as the key frame corresponding to each video clip. Alternatively, an image frame can be randomly selected from each video clip as the key frame corresponding to each video clip. It should be understood that the above embodiments are only exemplary, and the specific method of determining the key frame and the value of the frame rate r can be selected according to user needs, and this specification does not impose any restrictions on this.

[0061] Since the changes between consecutive frames of the target video may be very small, consecutive image frames usually contain a large amount of redundant information. Therefore, by dividing the target video into multiple video segments, determining a key frame sequence (first at least a part of the image frame) in the multiple video segments, and determining the spatial features based on the key frame sequence, compared to the method of determining the spatial features based on all consecutive frames of the target video, the amount of calculation in the process of determining the spatial features can be greatly reduced, thereby improving the calculation efficiency of the spatial features.

[0062] Still taking the example of determining the sub-spatial features corresponding to multiple sub-spatial dimensions based on the key frame sequence as an example, in some possible embodiments, the semantic features corresponding to the semantic dimension can be extracted by performing feature extraction on the key frame sequence through a contrastive language-image pre-training (CLIP) model. T-1 Each key frame in the} is input into the CLIP model in turn, and the quality perception features of CLIP for each key frame are output By taking the average of the quality perception features of all key frames, the quality perception feature f of the entire target video in the semantic dimension can be obtained: clip .

[0063] The distortion features corresponding to the distortion dimension may be extracted in a manner as follows: a video quality assessment model is pre-trained based on a large number of labeled image data sets, and the trained video quality assessment model can extract distortion features (including distortion type, distortion level, etc.) from the image. Each image in the image data set has a corresponding preset label, and the content of the preset label includes but is not limited to the distortion type (such as blur, noise, compression distortion, etc.) and its severity (level) of each image. In some possible embodiments, the model for extracting distortion features may select a contrastive learning for image quality estimation (CONTRASTIFF LEARNING FOR IMAGE QUALITY ESTIMATION, CONTRIQUE) model, and the CONTRIQUE model may be based on a key frame sequence {I0,I1,…,I T-1}, determine the distortion features corresponding to the key frame sequence Of course, the model for extracting distortion features can also select an image quality assessment model based on deep learning (Deep Image Structure and Texture Similarity, DISTS), etc. The selection of the specific model can be flexibly adjusted according to user needs and is not limited to the above embodiments. Videos with reduced quality often have distortions such as Gaussian blur and random noise. Therefore, effective identification and perception of various distortions in the spatial dimension of the target video will help to accurately evaluate the quality of the target video in the future.

[0064] The edge features corresponding to the edge dimension can be extracted by: using the edge detection algorithm (Canny) to extract edge information based on the key frame sequence, and further extracting the edge feature map through the pre-trained CNN network to obtain the edge features corresponding to the key frame sequence. First, separate each key frame image (red, green and blue three channels, RGB) in the key frame sequence in the channel dimension to obtain three single-channel images of red, green and blue corresponding to each key frame: R ,c G ,c B . Apply the Canny operator to the three single-channel images respectively to obtain the edge feature map corresponding to each single-channel image. Then use the Canny algorithm to determine the edge position by calculating the gradient strength of each pixel in the image. By setting two thresholds, the points with gradient strength higher than the high threshold are marked as edge points, and the points with gradient strength lower than the low threshold are marked as non-edge points. The three single-channel edge feature maps are spliced ​​together to form an image E containing the edge information of all channels. i (The stitched image contains the edge information of the keyframes in three color channels). The ⊕ represents the feature concatenation operation. The edge features corresponding to the key frame sequence are then further extracted through the pre-trained Convolutional Neural Network (CNN). It should be understood that the above embodiments are only exemplary, and the specific edge detection algorithm can also select Sobel operator, Prewitt operator, etc. The selection of specific detection algorithms and models can be flexibly adjusted according to user needs and is not limited to the above embodiments.

[0065] The comprehensive features corresponding to the comprehensive dimension can be extracted in a manner that converts the image quality scoring work into the generation of discrete quality grade vocabulary, which has good accuracy. In the method provided in this specification, a large multimodal model (Q-Align) can be used to determine the comprehensive features. The Q-Align model uses a large multimodal model to understand and analyze the content and context of each key frame image in the key frame sequence. Subsequently, the Q-Align model will evaluate the quality of the key frame image based on visual features (such as edges, textures, color distribution, etc.), high-level features (such as object recognition, scene classification, etc.), quality indicators (clarity, contrast, color saturation, etc.), image compression distortion, noise, etc. After evaluating the quality of the key frame image, the Q-Align model will convert these quality evaluation results into discrete quality grade vocabulary. That is, the Q-Align model will map the quality of the key frame image to a predetermined quality grade based on the evaluation results. For example, multiple quality grades can be defined, such as "excellent", "good", "general", "poor", etc. These levels can be determined by pre-set thresholds, such as by different quality index score intervals (for example, if the comprehensive score of the key frame image is 0.85, it can be mapped to "excellent"; a score of 0.6 can be mapped to "good", etc.). Finally, the Q-Align model will output the comprehensive features corresponding to the key frame sequence based on the key frame sequence. It should be understood that the above embodiments are only exemplary, and the specific multimodal model can also be selected from the Vision and Language Transformer (ViLT), the Adversarial Learning of Image and Text Representations (ALIGN), etc. The selection of the specific multimodal model can be flexibly adjusted according to user needs and is not limited to the above embodiments.

[0066] Still taking the sub-spatial domain dimensions including the above-mentioned semantic dimension, distortion dimension, edge dimension and comprehensive dimension as an example, the above-mentioned multiple sub-spatial domain features are dimensionally unified to obtain the spatial domain feature f IQA The method can be: IQA =ω(f flip )⊕ω(f edge )⊕ω(f dist )⊕ω(f Qalign ), where ω is a multi-layer perceptron (MLP), which processes the dimensional differences between different sub-spatial features through MLP, so that the processed multiple sub-spatial features can be combined in a unified dimension.

[0067] S350: Taking the image frame sequence as the granularity, extract features of the second at least part of the image frames in the target video to obtain time domain features and space-time features, wherein the time domain features represent the dynamic change information between the image frames in the target video, and the space-time features simultaneously represent the static information within a single image frame in the target video and the dynamic change information between the image frames.

[0068] Figure 5 FIG. 4 is a flow chart of a video quality detection method provided according to another embodiment of the present specification; Figure 5 In the method P300 provided in this specification, step S350 may include S351-S359:

[0069] S351: Generate a target frame sequence based on the second at least partial image frame.

[0070] In the embodiment of the present specification, the second at least partial image frame is a random selection of Q image frames in the target video according to a preset number Q, as the second at least partial image frame {m0, m1, ..., m Q-1}, the set consisting of the second at least partial image frames can also be called a target frame sequence.

[0071] Figure 6 The schematic diagram of the structure of the target video provided according to an embodiment of the present specification is shown in Figure 6. The target video includes T video clips, each of which includes multiple image frames. The star-shaped image frame is used to represent the first at least partial image frame (key frame), the circular image frame is used to represent the ordinary frame, and the rectangular image frame is used to represent the second at least partial image frame (target frame). It should be understood that Figure 6 This is merely an exemplary description, and the position of the specific key frame in the video clip and the position of the target frame can be adjusted according to user needs and the actual situation of the target video screen, and are not limited to those given in the above embodiments.

[0072] S353: Process the target frame sequence to obtain a first frame sequence.

[0073] In some possible embodiments, in order to reduce the computational efficiency of subsequent extraction of temporal features, each image frame in the target frame sequence may be directly scaled as a whole to obtain the first frame sequence I i {i0,i1,…,i Q-1 The overall scaling method may be: reducing the size of each image frame in the target frame sequence to a preset size, such as I i ∈R 224 ×224 , wherein the preset size is smaller than the original size of each image frame. That is, the size of each image frame in the first frame sequence is smaller than the size of each image frame in the target frame sequence; the resolution of each image frame in the first frame sequence is smaller than the resolution of each image frame in the target frame sequence. It should be understood that the above preset size is only an exemplary description, and of course it can also be any preset size such as 240*240, 160*160, etc., and this specification does not impose any limitation on this.

[0074] S355: Process the target frame sequence to obtain a second frame sequence.

[0075] The size of each image frame in the second frame sequence is smaller than the size of each image frame in the target frame sequence, and the resolution of each image frame in the second frame sequence is the same as the resolution of each image frame in the target frame sequence.

[0076] The method for determining the second frame sequence may be: for each image frame in the target frame sequence, dividing the image frame into a plurality of sub-blocks, extracting some sub-blocks from the plurality of sub-blocks as target sub-blocks, and splicing the target sub-blocks according to the relative positions between the target sub-blocks to obtain a spliced ​​image frame corresponding to the image frame; and generating the second frame sequence n based on the spliced ​​image frames corresponding to the image frames in the target frame sequence. i {n0,n1,…,n Q-1}.

[0077] For example, in the embodiments of this specification, the image frame can be divided into sub-blocks based on 16*16, and multiple sub-blocks are obtained, and the size of each sub-block is 16*16. Then, 14*14 sub-blocks are randomly selected from the multiple sub-blocks as target sub-blocks, wherein the size of each target sub-block is 16*16. And based on the relative positions between the target sub-blocks, the sub-blocks are spliced ​​to obtain a spliced ​​image frame corresponding to the image frame, and the size of the spliced ​​image frame is 244*244 (that is, n i ∈R 224×224). In the above process, although the size of the spliced ​​image frame (244*244) becomes smaller than the original size of the image frame (which may be 2048*2048, or 1280*1280, or 1920*1920, etc.), the resolution of each target sub-block contained in the spliced ​​image frame is not reduced, and part of the texture information and edge information contained in each target sub-block can be retained. Therefore, by reducing the size of the image frame as described above, not only can the amount of calculation for obtaining the spatiotemporal features based on the second frame sequence be reduced, but also the quality of the image in the temporal domain dimension and the quality of the image in the spatial domain dimension can be fully considered during the calculation process, thereby improving the accuracy of the calculation results.

[0078] S357: Extract features from the first frame sequence to obtain time domain features.

[0079] In an embodiment of the present specification, a first feature extraction network is used to perform feature extraction on a first frame sequence to obtain time domain features, and the first feature extraction network is trained to extract motion-related features in the frame sequence from a time domain dimension.

[0080] In some possible embodiments, the first feature extraction network may be an action recognition model (SlowFast). SlowFast is a model widely used for video action recognition. It uses a fast and slow dual path to simulate the M cells (responsible for quickly processing motion information) and P cells (responsible for processing static information) of the human eye, and can capture motion information in the video well. SlowFast runs in the slow path at a low frame rate to capture spatial semantics, and runs in the fast path at a high frame rate to capture motion with fine temporal resolution. In the embodiments of this specification, only the Fast path of the SlowFast model is used to obtain motion information in the first frame sequence (that is, the time domain feature f sf Of course, the first feature extraction network can also be an inflated 3D convolution network (Inflated 3D Convolutions, I3D), or a two-stream convolutional neural network (Two-StreamConvNets). The selection of the specific first feature network can be flexibly adjusted according to user needs and is not limited to the above embodiments.

[0081] S359: Extract features from the second frame sequence to obtain spatiotemporal features.

[0082] In an embodiment of the present specification, a second feature extraction network is used to perform feature extraction on the second frame sequence to obtain spatiotemporal features, and the second feature extraction network is trained to extract features from the frame sequence from both the temporal and spatial dimensions.

[0083] In the embodiments of this specification, in order to better perceive the video quality, it is necessary not only to extract temporal and spatial features, but also to combine temporal and spatial features to comprehensively evaluate the quality of the video. fast , which can more comprehensively capture the dynamic changes between image frames, as well as the correlation within and between frames.

[0084] In some possible embodiments, the second feature extraction network may be a Fast Video Quality Assessment (FAST-VQA) network, which uses gated relative position biases (GRPB) to effectively solve the problem that adjacent blocks in the sampled image frames are not adjacent in the original image frames due to random sampling and splicing. i As input, the global information, local information and temporal information of the image frames in the second frame sequence are output in the spatial dimension. In order to improve the accuracy of spatiotemporal feature extraction, the FAST-VQA model will be fine-tuned during the training process. The fine-tuned FAST-VQA model will be used to extract spatiotemporal features of the target video. fast Of course, the second feature extraction network can also be a video quality metric (VQM) model, or a video multi-method assessment fusion model (VMAF) model. It should be understood that the above embodiments are only exemplary, and the selection of the specific second feature extraction network can be flexibly adjusted according to user needs and is not limited to the above embodiments.

[0085] S370: Generate fusion features based on the spatial domain features, the temporal domain features and the spatiotemporal features, and determine the quality detection result of the target video based on the fusion features.

[0086] In some possible embodiments, the time domain features and the time and space features may be spliced ​​according to the weight information corresponding to the time domain features and the weight information corresponding to the time and space features to obtain an intermediate fusion feature; and a fusion feature f may be generated based on the intermediate fusion feature and the space domain feature. last .

[0087] Among them, since the dimensions of the time domain features and the space-time features are not necessarily the same, before splicing the time domain features and the space-time features, it is necessary to unify the dimensions of the time domain features and the space-time features. The way to unify the dimensions can be f VQA =ω(f sf )⊕ω(f fast), where ω is a multi-layer perceptron (MLP), which processes the dimensional differences between different time domain features and spatiotemporal features through MLP, so that the processed time domain features and spatiotemporal features can be combined in a unified dimension to obtain the combined intermediate fusion features.

[0088] Among them, the weight information corresponding to the spatiotemporal features and the weight information corresponding to the time domain features can be pre-trained by iteration. It is determined in advance based on a large number of video sample-quality score sets to indicate the importance of the time domain features or spatiotemporal features for subsequent video quality detection. By splicing the spatiotemporal features and the time domain features according to their respective corresponding weight information to obtain the intermediate fusion features, it can be ensured that the more important features eventually account for a larger proportion in the intermediate fusion features.

[0089] In some possible embodiments, in the process of generating fusion features according to the intermediate fusion features and the spatial features, the weight information corresponding to the intermediate fusion features and the weight information corresponding to the spatial features may also be considered. For example, based on the weight information corresponding to the intermediate fusion features and the weight information corresponding to the spatial features, the intermediate fusion features and the spatial features are concatenated to obtain the fusion features.

[0090] In some possible embodiments, the video quality detection method described in this specification may be implemented by a pre-trained video quality detection model. Figure 7 FIG. 4 shows a schematic diagram of the structure of the video quality detection model. Figure 7 As shown, the video quality detection model includes an IQA branch and a VQA branch. The IQA branch includes multiple sub-spatial feature extraction networks and an IQA fusion network, and the VQA branch includes a temporal feature extraction network, a spatiotemporal feature extraction network, and a VQA fusion network.

[0091] Still taking the example that the sub-spatial domain dimensions include: semantic dimension, distortion dimension, edge dimension and comprehensive dimension, the IQA branch includes: semantic feature extraction network, distortion feature extraction network, edge feature extraction network and comprehensive feature extraction network. Among them, the semantic feature extraction network is used to extract semantic features of the semantic dimension. The distortion feature extraction network is used to extract distortion features of the distortion dimension. The edge feature extraction network is used to extract edge features of the edge dimension. The comprehensive feature extraction network is used to extract comprehensive features of the comprehensive dimension. For example, the semantic feature extraction network can be a CLIP model, the distortion feature extraction network can be a CONTRIQUE model, and the edge feature extraction network can be a CNN model based on the Canny algorithm. The comprehensive feature extraction network can be a Q-Align model.

[0092] The temporal feature extraction network in the VQA branch can extract temporal features based on the second at least partial image frame in the target video. The spatiotemporal feature extraction network extracts spatiotemporal features based on the second at least partial image frame in the target video. For example, the temporal feature extraction network can be a SlowFast model, and the spatiotemporal feature extraction network can be a FAST-VQA model.

[0093] Continue to see Figure 7 In some embodiments, the IQA fusion network and the VQA fusion network may adopt a dual-branch structure. For example, the IQA fusion network and the VQA fusion network include a weight branch and a feature branch, respectively. In the IQA fusion network, the weight branch is used to calculate the importance of each sub-spatial domain feature obtained by different sub-spatial domain feature extraction networks for video quality detection, and obtain the weight information corresponding to different sub-spatial domain features. Subsequently, according to the weight information corresponding to each sub-spatial domain feature, each sub-spatial domain feature is spliced ​​to obtain the spatial domain feature.

[0094] The weight branch of the VQA fusion network is used to calculate the importance of the time domain features extracted by the time domain feature extraction network and the space-time features extracted by the space-time feature extraction network for video quality detection, and obtain the weight information corresponding to the time domain features and the space-time features. According to the weight information corresponding to the time domain features and the weight information corresponding to the space-time features, the time domain features and the space-time features are spliced ​​to obtain the intermediate fusion features. Among them, the weight information corresponding to each sub-spatial domain feature in the IQA fusion network, as well as the weight information corresponding to the time domain features and the weight information corresponding to the space-time features in the VQA fusion network will be continuously optimized and iterated according to the training of the video quality detection model to ensure the accuracy of the subsequent fusion features, and then ensure the accuracy of the video quality detection results of the subsequent target video.

[0095] Continue to see Figure 7 In the IQA fusion network, the weight information obtained by the weight branch is weighted and summed with the features obtained by each feature extraction network in the feature branch to obtain the spatial feature. In the VQA fusion network, the weight information obtained by the weight branch is weighted and summed with the features obtained by each feature extraction network in the feature branch to obtain the intermediate fusion feature. The fusion of the spatial feature and the intermediate fusion feature can obtain a fusion feature that is more suitable for video quality evaluation. For example, the fusion feature f last It can be expressed as:

[0096]

[0097] in, represents the weight branch in the IQA fusion network, represents the feature branch in the IQA fusion network, represents the weight branch in the VQA fusion network, It should be understood that the above embodiments are only exemplary, and the specific method of generating fusion features can be flexibly adjusted according to user needs and is not limited to the above embodiments.

[0098] Continue to see Figure 7 In the embodiments of the present specification, the video quality detection model may further include a regression network. The regression network may be a convolutional neural network (CNN), a recurrent neural network (RNN), or a long short-term memory network (LSTM), etc., which is used to predict the quality detection result of the target video based on the fusion feature. It should be understood that the above embodiments are only exemplary, and the selection of a specific regression network can be flexibly adjusted according to user needs and is not limited to the above embodiments. The fusion feature is input into the regression network so that the regression network is based on the fusion feature f last Determine the quality detection result of the target video. The quality detection result may be, for example, a quality score, or a quality grade, etc. The specific form of the quality detection result may be flexibly adjusted according to user needs and is not limited to the above embodiment.

[0099] This specification does not limit the training process of the video quality detection model. The following is an example of a possible training method.

[0100] For example, considering that the IQA branch extracts spatial features at the granularity of a single image frame, and in terms of the spatial feature extraction of a single image frame, there are large-scale data sets and feature extraction networks trained based on the above large-scale data sets. Therefore, in the embodiments of the present specification, the feature extraction network of the IQA branch can adopt a network model pre-trained based on a large-scale data set. For example, the semantic feature extraction network, distortion feature extraction network, edge feature extraction network, comprehensive feature extraction network and time domain feature extraction network in the video quality detection model can directly adopt the pre-trained network model. In this case, in the training process of the video quality detection model, it is only necessary to train the time domain feature extraction network, spatiotemporal feature extraction network, IQA fusion network, VQA fusion network and regression network.

[0101] The specific training process can be as follows: prepare a sample video set in advance, which includes N sample videos. Each sample video has a corresponding authenticity score (i.e., training label). For example, the authenticity scores corresponding to the N sample videos are expressed as y = {y1, y2, ..., y N}.

[0102] For each sample video, the sample video is preprocessed to obtain a first at least partial image frame and a second at least partial image frame corresponding to the sample video. The first at least partial image frame is input into the IQA branch, and the IQA branch obtains features of multiple sub-spatial domain dimensions based on the input first at least partial image frame. The features of multiple sub-spatial domain dimensions are input into the IQA fusion network to obtain spatial domain features. The second at least partial image frame is input into the VQA branch, and the VQA branch obtains time domain features and spatiotemporal features based on the input second at least partial image frame. The time domain features and spatiotemporal features are input into the VQA fusion network to obtain intermediate fusion features. Based on the intermediate fusion features and the time domain features, fusion features are obtained. Finally, the fusion features are input into the regression network to obtain the predicted score of the sample video. Assume that the predicted scores corresponding to N sample videos are expressed as {Q1, Q2,…, Q N}.

[0103] In this specification, the networks that need to be trained in the video quality detection model (time domain feature extraction network, spatiotemporal feature extraction network, IQA fusion network, VQA fusion network and regression network) can be iteratively trained based on the loss functions plcc-loss and rank-loss, wherein the determination methods of the above two loss functions can be respectively expressed as follows:

[0104] in represents the average authenticity score of N sample videos, represents the average of the predicted scores of N sample videos, y i Represents the true score of the i-th sample video.

[0105] where y j represents the true score of the jth sample video, e(y i ,y j ) is an operator. If y i ≥y j When the value of e is 1, otherwise when y i <y j When the value of e is -1.

[0106] The final loss function can be expressed as L total =L plcc +β*L rank , where β is a hyperparameter. To minimize the loss function L total For the training purpose, the parameters of the time domain feature extraction network, spatiotemporal feature extraction network, IQA fusion network, VQA fusion network and regression network in the video quality detection model are adjusted.

[0107] According to the above training method, multiple iterations of training are performed. total After converging to the preset interval, it is determined that the video quality detection model training is completed. The trained video quality detection model can be used to evaluate the quality of the target video.

[0108] In summary, in the video quality detection method and system 130 provided in this specification, after acquiring the target video, spatial features are extracted with a single image frame as the granularity. Time domain features and time features are extracted with an image frame sequence as the granularity. Fusion features are obtained based on time domain features, spatial features and spatiotemporal features to determine the video quality detection result of the target video based on the same features. This way of generating fusion features allows the fusion features to contain features of multiple different dimensions, which can more accurately express the quality of the video, making the quality detection result of the target video determined based on the fusion features more accurate. In addition, in the video quality detection method and system 130 provided in this specification, the spatial features are composed of multiple sub-spatial features. Since different features have different effects on subsequent quality detection, in the process of splicing the spatial features based on multiple sub-spatial features, the weight information corresponding to each of the multiple sub-spatial features is considered, and the multiple sub-spatial features are spliced ​​based on the above weight information to obtain the spatial features. Similarly, since the splicing of time domain features and space-time features has different effects on subsequent quality detection, this specification will form intermediate fusion features based on the weight information corresponding to the time domain features and the weight information corresponding to the space-time features. Finally, the fusion features are determined based on the spatial domain features and the intermediate fusion features. This way of determining the fusion features can ensure that important features will eventually account for a larger proportion of the fusion features, further improving the accuracy and reliability of the video quality detection results.

[0109] On the other hand, the present specification provides a computer-readable non-transitory storage medium storing at least one instruction set for performing executable instructions of a video quality detection method. When the at least one instruction set is executed by the processor, the at least one instruction set instructs the processor to implement the steps of the video quality detection method P300 of the present specification. In some possible implementations, various aspects of the present specification can also be implemented in the form of a program product, which includes a program code. When the program product is run on the system 130, the program code is used to enable the system 130 to perform the steps of the method P300 described in the present specification. The program product for implementing the above method can use a portable compact disk read-only memory (CD-ROM) to include the program code and can be run on the system 130. However, the program product of the present specification is not limited to this. In the present specification, the readable storage medium can be any tangible medium containing or storing a program, which can be used by the instruction execution system or used in combination with it. The program product can use any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media include: an electrical connection with one or more conductors, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. A computer-readable storage medium may include a data signal propagated in a baseband or as part of a carrier wave, wherein a readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The readable storage medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, device, or device. The program code contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the above. The program code for performing the operations of this specification can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" language or similar programming languages.

[0110] The above is a description of a specific embodiment of the present specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require a specific order or a continuous order to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0111] In summary, after reading this detailed disclosure, those skilled in the art will appreciate that the foregoing detailed disclosure may be presented only by way of example and may not be limiting. Although not explicitly stated herein, those skilled in the art will appreciate that this specification requires various reasonable changes, improvements, and modifications to the embodiments. These changes, improvements, and modifications are intended to be proposed by this specification and are within the spirit and scope of the exemplary embodiments of this specification.

[0112] In addition, certain terms in this specification have been used to describe embodiments of this specification. For example, "one embodiment", "an embodiment" and / or "some embodiments" mean that a particular feature, structure or characteristic described in conjunction with the embodiment may be included in at least one embodiment of this specification. Therefore, it can be emphasized and should be understood that two or more references to "an embodiment" or "an embodiment" or "an alternative embodiment" in various parts of this specification do not necessarily refer to the same embodiment. In addition, specific features, structures or characteristics may be appropriately combined in one or more embodiments of this specification.

[0113] It should be understood that in the foregoing description of the embodiments of this specification, in order to help understand a feature and for the purpose of simplifying this specification, this specification combines various features in a single embodiment, figure or its description. However, this does not mean that the combination of these features is necessary. When reading this specification, it is entirely possible for a person skilled in the art to mark out some of the devices as separate embodiments. In other words, the embodiments in this specification can also be understood as the integration of multiple secondary embodiments. And the content of each secondary embodiment is also valid when it is less than all the features of a single aforementioned disclosed embodiment.

[0114] Each patent, patent application, publication of patent applications, and other materials, such as articles, books, specifications, publications, documents, articles, etc., cited herein, except to the extent that it is inconsistent or conflicting with this document or that has a limiting effect on the broadest scope of the claims, may be incorporated herein by reference and used for all purposes now or hereafter associated with this document. In addition, in the event of any inconsistency or conflict between the description, definition, and / or use of a term in any material and the description, definition, and / or use of a term in this document, the term in this document shall prevail.

[0115] Finally, it should be understood that the embodiments of the application disclosed herein are explanations of the principles of the embodiments of this specification. Other modified embodiments are also within the scope of this specification. Therefore, the embodiments disclosed in this specification are only used as examples and not as limitations. Those skilled in the art can adopt alternative configurations according to the embodiments in this specification to implement the applications in this specification. Therefore, the embodiments of this specification are not limited to the embodiments accurately described in the application.

Claims

1. A video quality detection method, comprising: Obtaining a target video, wherein the target video includes a plurality of image frames; Taking a single image frame as a granularity, extracting features from a first at least part of the image frames in the target video to obtain spatial features, wherein the spatial features represent static information inside a single image frame in the target video; Taking the image frame sequence as the granularity, extracting features of the second at least part of the image frames in the target video to obtain time domain features and space-time features, wherein the time domain features represent dynamic change information between the image frames in the target video, and the space-time features simultaneously represent static information within a single image frame in the target video and dynamic change information between the image frames; as well as Based on the spatial domain features, the temporal domain features and the spatiotemporal features, a fusion feature is generated, and based on the fusion feature, a quality detection result of the target video is determined.

2. The method of claim 1, wherein: The step of extracting features from the second at least part of the image frames in the target video to obtain time domain features and spatiotemporal features includes: generating a target frame sequence based on the second at least partial image frame; Processing the target frame sequence to obtain a first frame sequence, wherein the size of each image frame in the first frame sequence is smaller than the size of each image frame in the target frame sequence, and the resolution of each image frame in the first frame sequence is smaller than the resolution of each image frame in the target frame sequence; Processing the target frame sequence to obtain a second frame sequence, wherein the size of each image frame in the second frame sequence is smaller than the size of each image frame in the target frame sequence, and the resolution of each image frame in the second frame sequence is the same as the resolution of each image frame in the target frame sequence; Extracting features from the first frame sequence to obtain the time domain features; and Perform feature extraction on the second frame sequence to obtain the spatiotemporal features.

3. The method of claim 2, wherein: The step of processing the target frame sequence to obtain a first frame sequence includes: An overall scaling operation is performed on each image frame in the target frame sequence to obtain the first frame sequence.

4. The method of claim 2, wherein: The step of processing the target frame sequence to obtain a second frame sequence includes: For each image frame in the target frame sequence, the image frame is divided into a plurality of sub-blocks, some sub-blocks are extracted from the plurality of sub-blocks as target sub-blocks, and the target sub-blocks are spliced ​​according to relative positions between the target sub-blocks to obtain a spliced ​​image frame corresponding to the image frame; and The second frame sequence is generated based on the stitched image frames corresponding to the image frames in the target frame sequence.

5. The method of claim 2, wherein: The extracting features from the first frame sequence to obtain the time domain features includes: A first feature extraction network is used to perform feature extraction on the first frame sequence to obtain the time domain feature, and the first feature extraction network is trained to extract motion-related features in the frame sequence from the time domain dimension.

6. The method of claim 2, wherein: The extracting features from the second frame sequence to obtain the spatiotemporal features includes: A second feature extraction network is used to extract features from the second frame sequence to obtain the spatiotemporal features, and the second feature extraction network is trained to extract features from the frame sequence from both the temporal and spatial dimensions.

7. The method of claim 1, wherein: The method further comprises: Dividing the target video into a plurality of video segments, and extracting at least one key frame from each of the plurality of video segments, to obtain the first at least partial image frame; and A portion of image frames is randomly selected in the target video as the second at least portion of image frames.

8. The method of claim 1, wherein: The step of extracting features from the first at least part of the image frames in the target video to obtain spatial features includes: Performing feature extraction on the first at least partial image frame in multiple sub-spatial domain dimensions respectively to obtain sub-spatial domain features corresponding to each of the multiple sub-spatial domain dimensions, wherein the multiple sub-spatial domain dimensions include at least two of the following: a semantic dimension, a distortion dimension, an edge dimension, or a comprehensive dimension; and The sub-spatial domain features corresponding to the multiple sub-spatial domain dimensions are concatenated to obtain the spatial domain features.

9. The method of claim 8, wherein: For any first sub-spatial domain dimension among the multiple sub-spatial domain dimensions, the sub-spatial domain feature corresponding to the first sub-spatial domain dimension is obtained in the following manner: In the first sub-spatial domain dimension, feature extraction is performed on each image frame in the first at least partial image frames to obtain frame features of each image frame in the first sub-spatial domain dimension; as well as The frame features of the first at least partial image frame in the first sub-spatial domain dimension are averaged to obtain the sub-spatial domain features corresponding to the first sub-spatial domain dimension.

10. The method of claim 8, wherein: The step of concatenating the sub-spatial features corresponding to the multiple sub-spatial dimensions to obtain the spatial features includes: According to the weight information corresponding to each of the multiple sub-spatial domain dimensions, the sub-spatial domain features corresponding to the multiple sub-spatial domain dimensions are spliced ​​to obtain the spatial domain features.

11. The method of claim 1, wherein: The generating a fusion feature based on the spatial domain feature, the temporal domain feature and the spatiotemporal feature includes: splicing the time domain features and the space-time features according to the weight information corresponding to each of the time domain features and the space-time features to obtain an intermediate fusion feature; and A fusion feature is generated based on the intermediate fusion feature and the spatial domain feature.

12. A video quality detection system, comprising: At least one storage medium storing at least one instruction set for performing video quality detection; as well as At least one processor is communicatively connected to the at least one storage medium, wherein the at least one processor reads the at least one instruction set when running, and executes the method as described in any one of claims 1 to 11 according to the instructions of the at least one instruction set.

Images